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Preface 


In  response  to  historically  high  cost  growth  in  the  acquisition  of  space 
systems,  the  Under  Secretary  of  the  Air  Force,  in  accordance  with 
National  Security  Space  (NSS)  Acquisition  Policy,1  directed  the  Air 
Force  acquisition  community  to  support  the  development  of  inde¬ 
pendent,  accurate,  and  timely  cost  analyses  to  make  the  acquisition 
of  NSS  systems  more  realistic  in  terms  of  estimated  costs.  In  turn, 
the  former  commander  of  Air  Force  Space  Command  (AFSPC),  Gen 
Lance  W.  Lord,  and  the  former  commander  of  the  Air  Force  Space 
and  Missile  Systems  Center  (SMC),  Lt  Gen  Michael  Hamel,  asked 
RAND  Project  AIR  FORCE  to  assess  cost-estimating  requirements 
and  capabilities  of  SMC  cost-estimating  organizations — as  well  as 
their  resources,  tools,  methods  and  processes — and  to  recommend 
an  enhanced  approach  to  cost  analysis  aimed  at  improving  cost¬ 
estimating  for  space  systems  and  increasing  the  understanding  of 
factors  that  influence  their  cost. 

The  study  was  sponsored  by  the  former  commander  of  SMC, 
General  Hamel.  The  project  technical  monitor  was  Col  Delane  Agui¬ 
lar,  SMC/FMC.  The  research  was  conducted  within  the  Resource 
Management  Program  of  RAND  Project  AIR  FORCE  as  part  of  a 
multiyear  study  entitled  “Air  Force  Space  Systems  Costs.”  The  initial 
data  collection  was  completed  in  May  of 2006  and  the  final  update  was 
provided  in  February  of  2007,  with  frequent  updates  in  between.  The 
final  briefing  was  presented  to  General  Hamel  on  March  13,  2007,  and 


1  National  Security  Space  Acquisition  Policy  (NSSAP),  Number  03-01,  December  20, 
2004. 
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to  Gen  Kevin  P.  Chilton,  the  former  commander  of  Air  Force  Space 
Command,  on  March  21,  2007. 

This  monograph  should  interest  government  personnel  involved 
in  cost  estimation  and  acquisition  of  defense  systems,  the  military 
space  acquisition  communities,  and  those  concerned  with  current  and 
future  acquisition  policies. 


RAND  Project  AIR  FORCE 

RAND  Project  AIR  FORCE  (PAF),  a  division  of  the  RAND  Corpo¬ 
ration,  is  the  U.S.  Air  Force’s  federally  funded  research  and  develop¬ 
ment  center  for  studies  and  analyses.  PAF  provides  the  Air  Force  with 
independent  analyses  of  policy  alternatives  affecting  the  development, 
employment,  combat  readiness,  and  support  of  current  and  future  aero¬ 
space  forces.  Research  is  conducted  in  four  programs:  Aerospace  Force 
Development;  Manpower,  Personnel,  and  Training;  Resource  Manage¬ 
ment;  and  Strategy  and  Doctrine. 

Additional  information  about  PAF  is  available  on  our  Web  site: 
http://www.rand.org/paf/ 
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Summary 


Background 

A  review  of  the  acquisition  programs  of  the  Department  of  Defense 
(DoD)  and  the  military  services  shows  that  they  have  a  history  of  cost 
growth.  This  is  especially  true  for  space  systems.  An  analysis  of  the 
data  contained  in  Selected  Acquisition  Reports  (SARs)  reported  from 
the  late  1960s  to  2004  shows  that  the  average  total  cost  growth  factor 
for  completed  Major  Defense  Acquisition  Programs  (MDAPs)  was  46 
percent.  This  percentage  was  calculated  by  comparing  the  actual  final 
acquisition  costs  of  a  program  to  its  cost  estimates  presented  in  the 
SAR  published  at  the  program’s  Milestone  B  decision  (MS  B)1  when 
the  program  was  approved  for  system  development  and  demonstra¬ 
tion  (SDD).  The  same  comparison  at  MS  C — the  program  approval  for 
production — reveals  that  cost  growth  had  not  been  eliminated.  In  fact, 
it  averaged  about  16  percent  for  all  MDAPs.  The  study  also  reveals  a 
systematic  bias  toward  underestimating  the  development  cost  for  space 


1  Most  DoD  weapon  systems  acquisition  programs  follow  guidelines  set  forth  in  U.  S. 
Department  of  Defense,  Department  of  Defense  Instruction  5000.2,  Subject:  Operation  of 
the  Defense  Acquisition  System,  May  12,  2003.  This  instruction  divides  the  defense  acquisi¬ 
tion  management  process  into  five  separate  milestones,  which  represent  key  decision  points; 
the  first  three  are  commonly  referred  to  by  their  letter  designations  A,  B,  and  C.  MS  B  is 
the  decision  point  to  transition  from  technology  development  to  system  development  and 
demonstration.  For  programs  approved  prior  to  2003,  we  used  the  equivalent  milestone 
designation. 
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systems;  this  underestimation  was  higher  than  the  underestimation  for 
other  weapon  system  types  included  in  the  analysis.2 

AFSPC  and  SMC  asked  PAF  to  examine  the  cost-estimating  pro¬ 
cess  for  some  existing  high-visibility  programs  and  to  provide  recom¬ 
mendations  based  on  lessons  from  these  programs.  Moreover,  RAND 
was  asked  to  assess  the  cost-estimating  requirements  and  capabilities  of 
SMC  cost-estimating  organizations — along  with  their  resources,  tools, 
methods,  and  processes — and  to  recommend  an  improved  approach  to 
cost  analysis. 

The  primary  mechanisms  PAF  used  to  carry  out  its  analysis  were 
an  in-depth  examination  of  two  projects  selected  by  the  Air  Force, 
the  Space  Based  Infrared  System  (SBIRS)-High  and  the  Global  Posi¬ 
tioning  System  (GPS);  extensive  interviews  with  all  the  space  System 
Program  Offices  (SPOs)  at  SMC;  discussions  with  other  agencies  and 
organizations  that  estimate  the  cost  of  acquiring  systems;  a  supply-and- 
demand  analysis  of  cost  analysis  personnel  workload  data  from  SMC; 
and  a  review  of  academic  literature  on  organizational  structure  and 
weapon-system  acquisition  documents. 


Conclusions 

Although  the  SBIRS-High  program  had  remarkably  stable  require¬ 
ments  from  1996  through  2005,  it  encountered  many  difficulties  keep¬ 
ing  to  its  planned  budget  and  schedule.  Our  analysis  shows  that  the 
SBIRS-High  program  experienced  high  cost  growth  of  300  to  350  per¬ 
cent  after  MS  B  and  that  most  of  this  growth  resulted  from  inappropri¬ 
ate  cost  and  schedule  estimates  made  by  the  contractor  and  accepted 
by  the  government. 

With  respect  to  GPS,  which  was  chosen  because  of  its  reputation 
as  a  well-managed  program,  our  analyses  indicate  that  while  the  pro- 


2  Mark  Arena,  Robert  S.  Leonard,  Sheila  E.  Murray,  and  Obaid  Younossi,  Historical  Cost 
Growth  of  Completed  Weapon  System  Programs ,  Santa  Monica,  Calif.:  RAND  Corporation, 
TR-343-AF,  2006;  and  Obaid  Younossi,  Mark  V.  Arena,  Robert  S.  Leonard,  Charles  Robert 
Roll,  Jr.,  Arvind  Jain,  and  Jerry  M.  Sollinger,  Is  Weapon  System  Cost  Growth  Increasing?  Santa. 
Monica,  Calif.:  RAND  Corporation,  MG-588-AF,  2006. 
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gram  had  an  aggregate  cost  underrun,  significant  components  of  that 
program  experienced  substantial  cost  growth.  Much  of  that  growth 
stemmed  from  cost-estimating  errors.  Also  worthy  of  note  is  the  fact 
that  the  program  requirements  changed  substantially. 

Both  programs  had  poor  results  from  the  portions  of  the  contracts 
that  were  awarded  under  the  Total  System  Performance  Responsibility 
approach.  Also,  both  programs  benefited  from  satellites  in  their  prede¬ 
cessor  programs  that  remained  operational  substantially  longer  than 
anticipated.  In  general,  both  programs  significantly  underestimated 
the  technical  risks  and  their  potential  effect  on  costs. 

Beyond  cost-estimating  errors  and  decisions  that  caused  pro¬ 
gram  changes,  the  research  team  identified  several  broad  problem  areas 
endemic  to  the  overall  SMC  cost-estimating  function  that  also  con¬ 
tributed  to  inaccurate  estimates.  The  downsizing  in  the  space  industry 
in  the  1990s  reduced  the  number  of  contractors  and  raised  the  stakes 
for  those  remaining  with  respect  to  obtaining  government  contracts. 
The  dwindling  contractor  base  fostered  a  sense  that  failure  to  win  one 
of  the  few  remaining  space  contracts  could  result  in  the  demise  of  the 
space  sector  of  the  company,  and  thus  contractors  were  tempted  to 
underbid  contracts  to  win  them.  This  all  occurred  at  a  time  when  space 
programs  were  becoming  increasingly  complex.  The  subsequent  turbu¬ 
lence  in  the  space  industry,  with  plant  closings,  mergers,  and  shifting 
contractor  personnel  from  one  place  to  another,  complicated  the  entire 
process  of  project  management. 

Furthermore,  the  acquisition  process  as  a  whole  fostered  optimis¬ 
tic  cost  estimates.  Institutional  and  budgetary  factors  tended  to  erode 
the  objectivity  of  the  cost-estimating  process,  particularly  the  lack  of 
independence  of  the  cost  analysts  from  the  program  offices.  Moreover, 
the  implementation  of  acquisition  reform  also  fostered  optimistic  esti¬ 
mates  and  eroded  the  government’s  ability  to  oversee  contractor  activi¬ 
ties.  The  organizational  structure  and  distribution  of  responsibilities 
contributed  to  problematic  estimates,  as  did  inadequate  numbers  of 
experienced  analysts  and  a  lack  of  relevant  data  and  methods  to  deal 
with  space  system  complexities.  Limited  cost,  programmatic,  techni¬ 
cal,  and  schedule  data — along  with  insufficient  coordination  among 
cost  analysts  and  engineers — created  problems  that  were  exacerbated 
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by  the  lack  of  adequate  risk- assessment  processes  and  methods,  includ¬ 
ing  independent  assessments  of  programmatic,  technical,  and  schedule 
assumptions. 

With  respect  to  assessing  the  cost  implications  of  technical  risk, 
both  the  SBIRS-High  and  GPS  programs  lacked  rigor.  In  part,  this 
was  due  to  the  effects  of  acquisition  reform  described  above,  but  it 
also  resulted  from  overreliance  on  contractor  capabilities.  Technical 
experts  focused  on  identifying  risk  in  specific  technologies  but  under¬ 
estimated  the  risk  associated  with  the  integration  effort  required  for 
a  complex  system.  Risk  assessments  were  not  always  made,  and  those 
that  were  sometimes  lacked  rigorous  fact-finding  to  support  the  assess¬ 
ments  of  technical  content.  Furthermore,  up-to-date  data  were  lack¬ 
ing  and  were  inaccurate  in  some  instances.  Also,  there  is  evidence  to 
suggest  that  some  risk  assessments  were  not  entirely  independent  and 
objective,  having  been  done  by  the  SPO,  by  the  prime  contractor,  or  by 
contracted  support  personnel.  Some  methodological  limitations,  such 
as  the  selection  of  cost-probability  distributions,  may  also  have  contrib¬ 
uted  to  estimating  errors.  Perhaps  the  biggest  single  challenge  for  cost 
estimation  in  these  programs  was  the  development  of  credible  method¬ 
ologies  for  determining  technical  risk,  quantifying  it,  and  incorporat¬ 
ing  the  risk  assessment  into  the  cost-estimating  process  to  produce  a 
credible  estimate. 

All  of  this  notwithstanding,  the  research  indicates  that  much  cost 
growth  falls  beyond  the  purview  of  the  cost-analysis  profession.  Costs 
increase  for  a  number  of  reasons,  some  of  which  are  avoidable  and  some 
of  which  cannot  be  avoided. 

A  considerable  portion  of  the  cost  analysis  is  done  by  systems 
engineering  and  technical  assistance  (SETA)  contractors,  who  appear 
to  carry  out  much  of  the  day-to-day  work  for  SMC.  Military  person¬ 
nel  have  excellent  cost  analysis  and  quantitative  skills,  but  they  typi¬ 
cally  rotate  out  after  one  assignment  and  rarely  serve  again  in  a  cost¬ 
estimating  position.  With  respect  to  demand  for  future  cost  analysts,  if 
we  assume  that  the  staff  can  accomplish  the  cost-estimating  tasks  more 
efficiently,  then  the  demand  for  cost  analysis  will  never  exceed  supply. 
If  the  staff  accomplishes  the  cost-estimating  tasks  less  efficiently,  then 
demand  will  exceed  supply  about  one-quarter  of  the  time.  But  assum- 
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ing  the  workforce  can  be  freely  assigned  to  where  it  is  most  needed, 
by  and  large,  SMC  had  an  adequately  sized  workforce  to  meet  its  pro¬ 
jected  demands,  except  for  one  peak  period  in  2007 — as  long  as  the 
future  portfolio  of  SMC  programs  remains  about  the  same  as  today 
in  terms  of  size  and  complexity.  Finally,  our  review  of  the  SMC  cost- 
analysis  organization  suggests  that  it  would  benefit  from  a  different 
organizational  structure. 


Recommendations 

Our  specific  recommendations  are  as  follows: 

Institute  Independent  Program  Reviews 

We  recommend  that  independent  teams  of  experts  work  along  with  cost 
estimators  to  perform  independent  reviews  in  conjunction  with  major 
program  reviews  and  milestones.  Mechanisms  or  processes  should  be 
developed  so  that  cost  analysts  can  draw  on  broader  SMC  technical 
expertise  as  resources  for  objective  and  independent  technical  and 
schedule  assessments,  the  two  key  factors  in  credible  cost  estimates.  We 
recommend  that  SMC’s  chief  engineer  be  required  to  review  and  coor¬ 
dinate  all  programmatic,  technical  baseline,  and  schedule  assumptions, 
as  contained  in  the  Cost  Analysis  Requirements  Description  (CARD). 
SMC  must  have  long-term  organizational  accountability  not  only  for 
cost  estimates,  but  also  for  programmatic,  technical,  schedule,  and  risk 
assessments.  (See  pp.  71-99  and  145-146.) 

Place  Special  Emphasis  on  Technical  Risk  Assessment 

Good  cost  estimates  hinge  on  accurate  technical  inputs.  Independent, 
rigorous  formal  technical  risk  assessments  are  needed  to  support  all 
cost  estimates  and  should  be  routinely  updated.  All  cost  and  technical 
risk  assessments  should  be  cross-checked  using  alternative  methodolo¬ 
gies  (e.g.,  historical  analogies  compared  with  parametric  analyses).  The 
quality  of  the  inputs  to  the  technical  assessments  should  be  improved 
by  collecting  and  making  available  more  relevant  data  and  increasing 
visibility  into  contractor’s  capabilities.  The  level  of  technical  expertise 
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and  the  communications  among  technical,  program,  and  cost  experts 
should  be  enhanced.  (See  pp.  45-69.) 

Adopt  a  Hybrid  Cost  Organizational  Structure  for  SMC 

A  hybrid  structure,  which  includes  the  strengths  of  both  centralized 
and  decentralized  organizations,  has  the  most  potential  benefits  and 
the  fewest  limitations.  In  particular,  increasing  the  independence  of 
the  analysts  performing  major  cost  estimates  will  improve  the  reliabil¬ 
ity  of  the  estimates  and  SMC’s  reputation  as  an  organization  whose  cost 
numbers  can  be  trusted.  This  change  will  require  significant  support 
from  senior  SMC  leadership,  as  we  discuss  below.  (See  pp.  127-139.) 

Realign  and  Strengthen  the  Future  Financial  Management  (FM) 
Organization  by  Reassigning  Cost-Estimating  Tasks 

We  recommend  that  the  cost  tasks  be  divided  between  cost  staff  in 
the  comptroller  organization  and  the  program  offices.  Cost-estimating 
tasks  should  be  done  within  the  SPO  when  the  focus  is  on  program 
execution,  where  changing  priorities  or  rapid  responses  are  common, 
for  functions  required  to  manage  the  day-to-day  activities  of  the  pro¬ 
gram,  where  the  official  position  for  effective  interaction  with  SPO  per¬ 
sonnel  is  needed  and  where  processes  are  unique  to  the  program.  The 
comptroller’s  cost  staff  should  perform  the  tasks  when  independent 
analysis  is  a  priority,  experienced  government  leadership  is  required, 
economies  of  scale  exist,  flexibility  in  assignments  is  desired,  skill  sets 
and  tasks  fall  outside  the  SPO  mission,  and  workload  and  priorities  are 
generally  predictable.  (See  pp.  128-130  and  140-141.) 

Require  Major  Estimates  to  Be  Led  by  Experienced  and  Qualified 
Government  Analysts 

Contractor  support  staff  should  not  lead  major  cost  estimates.  How¬ 
ever,  contractor  support  plays  an  important  role  in  data  collection, 
building  cost  models,  documenting  the  results,  and  other  technical 
assistance.  SMC  and  Air  Force  human  resources  organizations  will 
need  to  support  the  new  staffing  approach.  The  current  approach  to 
hiring,  personnel  assignments,  civil  service  grade  structure,  and  mili¬ 
tary  force  development  regulations  may  need  to  be  reassessed  to  attract 
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and  retain  competent  cost  analysts  in  SMC.  Furthermore,  we  note  that 
a  few  experienced  analysts  can  be  more  effective  than  many  inexperi¬ 
enced  ones.  (See  pp.  130  and  138-141.) 

Implement  Best  Practices  from  Other  Cost  Organizations 

Our  team  met  with  various  organizations  performing  cost  analy¬ 
sis  and  collected  best  practices.  Interviewees  overwhelmingly  agreed 
that  sound  initial  estimates  are  critical  and  should  be  appropriately 
resourced.  Other  widely  supported  best  practices  consist  of 

•  including  analysts  with  technical/engineering,  fmancial/busi- 
ness  management,  economics,  mathematics,  and  statistics  back¬ 
grounds  in  cost-estimating  teams 

•  updating  annual  program  cost  and  risk  estimates 

•  keeping  a  track  record  of  each  estimate 

•  reviewing  and  archiving  all  major  estimates 

•  emphasizing  monthly  Earned  Value  Management  analysis  as  a 
management  tool.  (See  pp.  131-133  and  159-168.) 

Standardize  Cost  Data  Collection  and  Improve  Current  Databases 

In  addition  to  historical  cost  information,  the  SMC  Comptroller’s 
cost  staff  should  also  collect  historical  programmatic,  technical,  and 
schedule  data  and  archive  it  for  future  use.  We  encourage  regular  data 
exchanges  with  internal  Air  Force  organizations,  such  as  the  Air  Force 
Cost  Analysis  Agency,  and  external  organizations,  such  as  the  National 
Reconnaissance  Office  and  NASA,  as  a  critical  aspect  of  this  data  col¬ 
lection.  (See  pp.  193-202.) 
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Introduction 


Background 

Previous  work  by  RAND  Project  AIR  FORCE  (PAF)  concluded  that 
the  Department  of  Defense  (DoD)  and  the  military  departments  have 
historically  underestimated  the  cost  of  new  weapon  systems.  Analysis 
of  the  data  contained  in  Selected  Acquisition  Reports  (SARs),  reported 
between  the  late  1960s  to  2004,  showed  that  the  average  total  cost 
growth  for  completed  major  defense  acquisition  programs  (MDAPs) 
(adjusting  for  procurement  quantity  changes)  was  46  percent  from  the 
Milestone  (MS)  B  estimate  and  16  percent  from  the  MS  C  decision 
point  when  the  estimate  in  the  final  SAR  is  compared  to  the  respec¬ 
tive  estimates  at  each  milestone.  This  study  also  revealed  a  systematic 
bias  toward  underestimating  the  cost  for  space  systems,  which  was 
higher  than  the  costs  for  other  weapon  system  types  included  in  the 
analysis.1 

A  companion  study  on  the  trend  of  cost  growth  over  the  past  three 
decades  also  showed  that  development  cost  growth  for  that  period  has 
not  improved,  despite  many  attempts  to  reform  the  acquisition  pro- 
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cess.2  We  emphasize  here  that  the  cost  estimates  reported  in  the  SAR 
do  not  reflect  a  specific  cost  estimate  performed  by  the  cost-estimating 
functions  within  the  DoD;  rather,  they  are  the  department’s  President’s 
Budget  position.  In  some  cases,  the  cost  estimate  may  differ  substan¬ 
tially  from  the  budget  request. 

Table  1.1  shows  the  average  development  cost  growth  factor 
(DCGF)  five  years  after  the  MS  B  decision.  The  DCGF  is  defined  as 
the  ratio  of  the  cost  estimate  reported  in  the  SAR  dated  about  five  years 
after  the  MS  B  date  to  the  cost  estimate  reported  in  the  SAR  at  MS  B. 
The  mean  values  are  between  1.16  and  1.92  and  the  values  of  1.64  and 
1.91  correspond  to  satellites  and  launch  vehicles  respectively.  Similarly, 
median  values  are  between  1.13  and  1.88,  and  the  highest  value  of  1.88 
corresponds  to  the  DCGF  of  satellites. 

Figure  1.1  shows  the  total  annual  acquisition  budgets  in  then- 
year  dollars  (TY$)  for  the  nine  largest  ongoing  space  programs 


Table  1.1 

Average  DCGF  at  Five  Years  After  MS  B,  by  Program  Type 


Program  Type 

N 

Mean 

(Standard  Deviation) 

Median 

All  programs 

76 

1.45  (0.80) 

1.22 

Aircraft 

15 

1.16  (0.16) 

1.13 

Cruise  missiles 

5 

1.75  (0.95) 

1.43 

Electronic  aircraft 

5 

1.59  (0.31) 

1.65 

Electronics 

19 

1.20  (0.22) 

1.22 

Helicopters 

8 

1.92  (1.48) 

1.58 

Launch  vehicles 

3 

1.91  (1.53) 

1.15 

Missiles 

14 

1.50  (1.04) 

1.30 

Other 

1 

1.25 

1.25 

Satellites 

3 

1.64  (0.50) 

1.88 

Vehicles 

3 

1.81  (1.06) 

1.21 

SOURCE:  Younossi  et  al.  (2006). 


2  Obaid  Younossi,  Mark  V.  Arena,  Robert  S.  Leonard,  Charles  Robert  Roll,  Jr.,  Arvind 
Jain,  and  Jerry  M.  Sollinger,  Is  Weapon  System  Cost  Growth  Increasing: ?  Santa  Monica,  Calif.: 
RAND  Corporation,  MG-588-AF,  2006. 
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Figure  1.1 

Total  Air  Force  Space  Acquisition  Annual  Budgets  in  TY$,  2001-2007,  and 
Size  of  Largest  Programs 
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at  SMC  from  2001  through  2007,  compared  with  each  other  and  with 
the  annual  budget  value  of  all  other  acquisition  programs  at  SMC. 
There  was  a  steady  increase  between  2001  and  2007,  with  the  exception 
of  a  slight  decline  in  2005.  The  figure  also  shows  that  the  nine  largest 
programs  made  up  well  over  two-thirds  of  the  total  SMC  space  system 
acquisition  budgets  during  this  period. 

In  response  to  this  high  cost  growth  in  the  acquisition  of 
space  systems,  the  Under  Secretary  of  the  Air  Force,  in  accordance 
with  National  Security  Space  (NSS)  Acquisition  Policy3  directed 
the  Air  Force  to  support  the  development  of  independent,  accu¬ 
rate,  and  timely  cost  analyses  to  make  the  acquisition  of  NSS  sys- 


3  National  Security  Space  Acquisition  Policy  (NSSAP),  Number  03-01,  December  20, 
2004. 
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terns  more  realistic  in  terms  of  estimated  costs.  In  turn,  the  former 
Air  Force  Space  Command  Commander  (AFSPC/CC)  General 
Lance  Lord  and  the  Air  Force  Space  and  Missile  Systems  Center 
Commander  (SMC/CC)  Lt.  General  Michael  Hamel  asked  PAF  to 
assess  the  cost-estimating  requirements  and  capabilities  of  SMC  cost¬ 
estimating  organizations,  as  well  as  their  resources,  tools,  methods  and 
processes,  and  to  recommend  an  enhanced  approach  to  cost  analysis 
aimed  at  improving  cost  estimation  for  space  systems  and  increasing 
the  understanding  of  factors  that  influence  their  cost. 


Purpose 

The  purpose  of  this  project  was  to  conduct  research  and  analysis  that 
would  lead  to  recommendations  for  changes  in  SMC  organization,  per¬ 
sonnel,  methods,  and  processes  that  would  result  in  more  credible  and 
realistic  cost  estimates  of  space  systems. 


Methodology 

Our  general  objectives  were  to  determine  the  root  causes  of  the  high 
cost  growth  experienced  by  space  systems  at  SMC  and  to  provide  SMC 
with  specific  recommendations  and  suggestions  for  improvement.  Our 
research  approach  combined  qualitative  and  quantitative  methods  and 
relied  on  analysis  of  both  primary  and  secondary  data  sources.  Specifi¬ 
cally,  we  interviewed  111  people,  recategorized  cost  data  in  the  SARs  to 
better  incorporate  program  history,  developed  a  model  of  the  demand 
for  cost  analysis  personnel,  surveyed  program  offices  for  a  current  snap¬ 
shot  of  personnel  and  workload,  and  reviewed  internal  government 
documents  as  well  as  published  reports.  We  conducted  three  differ¬ 
ent  types  of  interviews:  one  set  that  contributed  to  our  case  studies, 
one  set  that  provided  additional  SMC-related  insights,  and  a  third  set 
that  involved  cost  analysis  personnel  from  outside  organizations.  Table 
1.2  lists  both  the  types  of  organizations  and  the  types  and  number  of 
people  interviewed  within  each  interview  category. 
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Table  1.2 

Organizations  and  the  Number  of  Interviewees 


Interview  Type 

Number 

Case  studies 

55 

SBIRS-High 

17 

GPS 

10 

Personnel  who  worked  on  both 
SBIRS-High  and  GPS  issues 

28 

SMC  Interviews 

42 

Program  managers;  deputy  program 
managers;  directors 

3 

Program  control  chiefs;  cost  analysts; 
earned  value  analysts 

39 

Other  cost  analysis  organizations 

14 

Air  Force  agencies 

9 

Other  government  agencies 

5 

Total 

111 

Case  Study  Analysis 

Our  case  studies  featured  an  in-depth  analysis  of  the  cost-estimating 
process  of  two  programs  selected  by  SMC:  the  Space  Based  Infrared 
System  (SBIRS)-High  program  and  the  Global  Positioning  System 
(GPS).  In  part,  this  analysis  involved  a  quantitative  analysis  of  the 
Selected  Acquisition  Reports  (SARs)  and  a  recategorization  of  the 
reported  cost  variance  data  into  a  consistent  RAND-developed  tax¬ 
onomy  that  allowed  an  easy  and  consistent  comparison  of  the  causes  of 
cost  growth  across  programs.  Another  part  of  the  analysis,  our  quali¬ 
tative  approach,  was  based  on  semi-structured  interviews  of  55  senior 
program  officials,  cost  analysts,  engineers,  and  others  during  34  sepa¬ 
rate  interview  sessions.4  We  also  reviewed  available  internal  documents 
and  published  reports. 

During  the  interviews,  we  asked  each  interviewee  to  explain 
(1)  his  or  her  beliefs  about  the  reasons  for  cost  growth,  (2)  details  of 


4  For  more  detail  discussion  of  unstructured  and  semi-structured  interviews,  see  Bernard 
H.  Russell,  Research  Methods  in  Anthropology:  Qualitative  and  Quantitative  Approaches ,  3rd 
ed.,  Walnut  Creek,  Calif.:  Altamira  Press,  2002. 
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how  the  cost-estimation  process  was  conducted  during  various  points 
in  the  program,  (3)  how  risk  analysis  assessments  were  conducted  and 
integrated  into  the  cost-estimating  process,  and  (4)  “lessons  learned” 
from  the  program  cost-estimating  process  and  how  it  might  have  been 
done  better.  A  minimum  of  two  to  a  maximum  of  four  RAND  proj¬ 
ect  team  members  conducted  each  interview.  Each  took  detailed  and 
extensive  interview  notes.  Generally,  one  person  later  transcribed  the 
notes;  the  notes  were  then  circulated  to  the  other  participants  for  cor¬ 
rections,  additions,  or  further  comments.  Occasionally,  the  notes  were 
sent  back  for  review  by  the  interviewee.  Only  those  issues  that  the 
study  team  agreed  were  corroborated  by  several  individuals  and  sup¬ 
ported  by  internal  documents  and  published  reports  are  documented 
in  this  monograph.  We  report  the  results  of  our  case  study  analysis  in 
Chapters  Two,  Three,  and  Four. 

SMC  Interviews  and  Workload  Analysis 

We  also  interviewed  42  members  of  SMC  System  Program 
Offices  (SPOs)  and,  through  a  survey,  collected  data  on  cost- 
analysis  personnel  and  workload.  The  individuals  we  inter¬ 
viewed  included  managers  and  cost  and  technical  analysts  from 
both  government  and  contractors  hired  to  support  the  day-to- 
day  function  of  an  SMC  program  office.  We  relied  on  both  semi- 
structured  and  unstructured  interviews.  We  provided  the  questions 
listed  in  Appendix  A  in  advance  of  interviews  to  provide  context  and 
help  the  interviewees  prepare.  These  questions  were  also  given  to  vari¬ 
ous  SMC  organizations  before  the  interviews  to  guide  our  discussions 
and  ensure  consistent  treatment  of  all  programs.  However,  during  the 
interviews  the  RAND  project  team  asked  other  open-ended  questions 
to  cover  issues  unique  to  each  program.  As  in  the  case  study  analysis, 
one  person  generally  transcribed  the  interview  notes,  which  were  then 
circulated  to  the  other  team  members  for  corrections,  additions,  or  fur¬ 
ther  comments.  Here,  our  focus  was  on  whether  systemic  problems 
existed  at  SMC  or  whether  the  issues  were  unique  to  the  case  study 
programs.  Appendix  B  presents  Endings  from  interviews  with  SMC 
program  offices  that  dealt  with  issues  other  than  those  specifically  sup¬ 
porting  the  separate  case  studies. 
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We  also  developed  a  model  to  analyze  current  and  future  cost 
analysis  personnel  requirements,  incorporating  survey  data  from  all 
current  SMC  programs.  The  data  were  derived  from  two  survey  instru¬ 
ments:  the  first  asked  cost  analysis  functional  leaders  to  identify  all 
military,  civil  service,  and  contractor  cost  and  earned  value  analysts;  to 
detail  their  acquisition,  cost,  and  earned  value  experience;  and  to  sum¬ 
marize  their  level  of  education,  Acquisition  Professional  Development 
Program  (APDP)  qualification,  and  pay  grade.  The  second  instrument 
asked  unit  leaders  to  assess  their  entire  workload,  broken  down  into 
nearly  70  tasks;  the  responses  assessed  how  frequently  each  task  has  to 
be  performed,  how  many  analysts  are  required,  and  how  much  time 
each  analyst  needs  to  complete  his  or  her  portion  of  the  task. 

A  time  line  of  survey  responses  can  be  found  in  Table  1.3.  For  each 
office,  type  of  survey,  and  date,  the  version  number  of  the  response  is 
listed.  Chapter  Five  presents  the  results  of  this  modeling  effort. 

Cost  Analysis  "Best  Practices" 

To  understand  how  a  successful  cost-estimating  organization  should 
use  cost  analysis,  the  research  team  interviewed  14  directors  and  tech¬ 
nical  directors  of  several  government  cost-analysis  organizations  out- 


Table  1.3 

Survey  Responses 


Supply  Survey 

Demand  Survey 

Office 

May 

2006 

October 

2006 

February 

2007 

October 

2006 

February 

2007 

GPS 

1 

2 

2 

1 

1 

EELV  (LR) 

1 

1 

3,4 

1 

2 

SBIRS-High 

1 

2 

3 

1 

1 

MILSATCOM 

1 

1 

1 

1 

2,3 

Space  Radar 

1 

2 

2 

1 

1 

SMC/FMC 

1 

2 

2 

1 

1 

NOTES:  EELV  (LR)=  Evolved  Expendable  Launch  Vehicle  (part  of  the  Launch 
and  Range  Systems  Wing);  MILSATCOM  =  military  satellite  communication; 
FMC  =  Cost  Directorate. 
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side  of  SMC  who  are  responsible  for  performing  similar  cost-analysis 
functions.  Those  organizations  include  the  Air  Force  Material  Com¬ 
mand  (AFMC)  Aeronautical  Systems  Center  (ASC),  AFMC  Electronic 
Systems  Center  (ESC),  Air  Force  Cost  Analysis  Agency  (AFCAA), 
National  Reconnaissance  Office  (NRO),  Naval  Air  Systems  Command 
(NAVAIR),  Naval  Sea  Systems  Command  (NAVSEA),  Office  of  the 
Secretary  of  Defense  (OSD),  and  the  National  Aeronautics  and  Space 
Administration  (NASA).  The  interviews  are  summarized  in  Appen¬ 
dix  B. 

We  also  reviewed  the  literature  on  space  program  histories, 
assessed  the  organizational  and  reporting  structures  of  cost  functions, 
and  looked  at  cost-estimating  data,  methods,  and  processes. 

The  results  of  these  interviews  and  document  reviews  were  com¬ 
pared  with  the  findings  of  the  case  studies  and  the  SMC  interviews  to 
form  the  basis  for  our  “best  practice”  recommendations.  Chapters  Six 
and  Seven  present  the  results  of  this  effort. 


Organization  of  Monograph 

The  monograph  contains  seven  chapters  and  five  appendixes.  Chapter 
Two  presents  the  two  case  studies  that  the  research  team  used  as  its 
analytic  point  of  departure,  on  SBIRS-High  and  GPS,  as  well  as  the 
results  of  the  interviews  with  SMC  cost  and  program  management  per¬ 
sonnel.  Chapter  Three  discusses  the  technical  risk-assessment  process 
related  to  cost  estimation  on  the  two  case  study  programs.  Chapter 
Four  describes  the  nontechnical  factors  that  contributed  to  the  cost 
variances  for  the  two  case  study  programs,  including  institutional  pres¬ 
sures,  the  acquisition  environment  at  the  time,  and  acquisition  reform. 
Chapter  Five  analyzes  the  SMC  workforce.  Chapter  Six  presents  our 
analysis  of  alternative  ways  of  organizing  SMC,  and  Chapter  Seven 
presents  the  study’s  conclusions  and  recommendations. 

The  first  of  the  five  appendixes  contains  the  questionnaire 
researchers  used  to  guide  their  interviews.  The  second  provides  a  sum¬ 
mary  of  interviews  of  SMC  programs  other  than  our  two  case  studies. 
The  third  lists  the  programs  managed  by  SMC.  The  fourth  discusses 
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the  details  of  our  cost  analysis  workload  and  personnel  model,  and 
the  fifth  describes  the  cost-estimating  process  tools,  methods,  and  data 
assessment  for  space  systems. 


CHAPTER  TWO 


Analysis  of  the  Magnitude  and  Sources  of  Cost 
Growth  in  the  SBIRS-High  and  GPS  Programs 


Introduction 

An  understanding  of  how  and  why  costs  have  grown  over  time  in  the 
SBIRS-High  and  GPS  programs  is  essential  to  provide  context  for  our 
findings  and  recommendations.  The  first  step  is  to  determine  a  base¬ 
line  from  which  the  costs  of  the  programs  are  tracked.  Total  expected 
acquisition  expenditures  for  weapon  systems  are  initially  estimated  at 
the  time  of  commitment  to  major  development  activities.  Within  the 
DoD  acquisition  system,  this  typically  occurs  at  MS  B.  The  estimate 
developed  in  support  of  this  milestone  provides  budget  guidance  and 
sets  expectations  regarding  the  funding  required  throughout  system 
acquisition.  Changes  to  this  estimate,  regardless  of  which  future  year’s 
budgets  are  affected,  disrupt  DoD’s  financial  management.  The  depart¬ 
ment  is  well  served  when  these  disruptions  are  minimized,  which  is 
best  accomplished  by  program  acquisition  estimates  at  MS  B  that  accu¬ 
rately  project  the  weapon  system’s  ultimate  acquisition  cost. 

In  the  vast  majority  of  MDAPs,  the  cost  of  the  weapon  system 
increases  after  MS  B.1  Much  of  this  cost  growth  is  beyond  the  pur¬ 
view  of  the  cost  analysis  profession  at  the  time  of  committing  to  major 


1  See  Younossi  et  al.,  2006;  Arena,  Leonard,  et  al.,  2006;  David  McNicol,  “Growth  in  the 
Costs  of  Major  Weapon  Procurement  Programs,”  IDA  Paper  P-3832,  2004;  and  Jeffrey  A. 
Drezner,  Jeanne  M.  Jarvaise,  Ron  Hess,  Daniel  M.  Norton,  and  Paul  G.  Hough,  An  Analy¬ 
sis  of  Weapon  System  Cost  Growth,  Santa  Monica,  Calif.:  RAND  Corporation,  MR-291-AF, 
1993. 
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development  in  space  systems.  Cost  increases  result  from  numerous 
causes,  some  of  which  are  avoidable  and  others  of  which  either  cannot 
or  should  not  be  avoided.  If  the  cost  estimate  at  MS  B  were  “perfected,” 
all  cost  growth  would  still  not  be  avoided.  To  isolate  that  portion  of 
cost  growth  that  can  reasonably  be  avoided  through  improvements  to 
cost-estimating  capability  at  SMC,  RAND  applied  its  cost-variance 
categorization  methodology  to  the  cost  variances  identified  in  the  1996 
through  2005  SARs  for  our  two  case  study  programs.2 

The  next  step  in  describing  the  cost  growth  history  of  these  pro¬ 
grams  is  to  define  boundaries  for  our  analyses.  This  is  not  difficult  in 
the  case  of  the  SBIRS  program,  which  began  in  earnest  in  Novem¬ 
ber  1996  with  the  declaration  of  MS  B  and  award  of  the  development 
contract.  For  the  GPS  program,  setting  the  boundaries  is  more  com¬ 
plex  because  the  history  of  the  program  is  more  complicated.  Here,  we 
briefly  describe  the  key  aspects  that  drove  our  choice  of  boundaries;  a 
more  detailed  explanation  of  the  program’s  history  appears  in  Chapter 
Three. 

The  GPS  program  has  a  long  history  dating  back  to  the  1970s, 
and  it  has  produced  several  different  versions  of  new  GPS  satellites. 
Development  contracts  were  awarded  for  replacement  IIF  series  satel¬ 
lites  and  the  modernization  of  the  operational  control  segment  (OCS) 
in  April  1996.  At  that  time,  the  HR  satellites  were  in  production.  These 
three  efforts  represented  the  vast  majority  of  GPS  acquisition  efforts 
(by  value)  under  way  in  1996  through  2005,  as  reported  in  the  SARs. 
April  1996  marked  the  beginning  of  major  development  on  two  of  the 
three  pieces  of  the  program;  those  are  the  two  pieces  we  chose  to  track 
in  detail.  Selecting  estimates  in  the  1996  time  frame  as  our  baseline 
for  both  programs  provides  the  additional  advantage  of  having  similar 
time  lines  for  both  programs. 


2  See  Joseph  G.  Bolten,  Robert  S.  Leonard,  Mark  V.  Arena,  Obaid  Younossi,  and  Jerry  M. 
Sollinger,  Sources  of  Weapon  System  Cost  Growth:  Analysis  of  35  Major  Defense  Acquisition 
Programs,  Santa  Monica,  Calif.:  RAND  Corporation,  MG-670-AF,  2008. 
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Categorization  and  the  Magnitude  of  Cost  Variances3 

The  RAND  cost-variance  categorization  follows  the  methodology  from 
a  RAND  PAF  study  entitled  “Sources  of  Cost  Growth.”4  Using  the 
data  from  the  SARs  of  35  MDAPs,  that  study  developed  cost  growth 
categories  oriented  toward  the  causes  of  cost  growth.  The  categories 
apply  to  all  four  acquisition-funding  categories:  development,  procure¬ 
ment,  acquisition-related  operations  and  maintenance  (O&M),  and 
military  construction  (MilCon).  Table  2.1  briefly  summarizes  the  cost- 
variance  categories. 

The  primary  aim  of  the  methodology  is  to  separate  “Errors”  from 
“Decisions.”  Errors  are  generally  defined  as  inaccurate  initial  esti¬ 
mates  of  cost,  schedule,  and  technology  development  to  accomplish 
the  original  work  scope  and  to  meet  original  capabilities  as  defined  at 
MS  B.  This  includes  difficulties  caused  by  overconfidence  in  the  matu¬ 
rity  of  relevant  technologies  and  overoptimistic  efficiency  expectations 
from  the  companies  contracted  to  design,  build,  and  test  the  systems. 
Decisions  involve  program  changes  within  the  control  of  an  entity  of 
authority — program  office,  SMC,  Air  Force,  OSD,  Congress,  and  the 
President.  They  include  adding,  removing,  and  changing  requirements; 
acquisition  strategy  changes  including  quantity,  rate,  and  contracting 
strategy;  fundamental  program  content  change;  and  affordability  or 
priority  changes  at  levels  above  the  program  that  add  or  remove  fund¬ 
ing  without  changing  requirements. 

The  approach  of  distinguishing  between  errors  and  deci¬ 
sions  is  consistent  with  a  similar  methodology  previously  devel¬ 
oped  by  the  Institute  for  Defense  Analysis.5  In  addition,  the  RAND 
methodology  has  been  vetted  and  approved  by  the  Office  of  the 
Deputy  Assistant  Secretary  for  Cost  and  Economics.  The  assign¬ 
ment  of  cost  variances  into  categories  and  subcategories  based 


3  We  use  the  term  cost  variance  instead  of  cost  growth  to  cover  both  cost  growth  (overrun) 
and  cost  underrun. 

4  This  study  is  detailed  in  Bolten  et  ah,  2008. 

5  See  McNicol,  2004. 
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Table  2.1 

Summary  Definitions  of  RAND  Cost-Variance  Categories 


Category 

Definition 

Errors 

Cost  estimation 

Program  rebudgeting  caused  by  an  inappropriate  initial 
estimate  of  costs 

Schedule  estimation 

Program  rebudgeting  and  rescheduling  caused  by  an 
inappropriate  schedule  plan 

Technical  issues 

Program  replanning  and  rebudgeting  resulting  from 
significant  technology  development  or  implementation 
problems 

Decisions 

Requirements 

Changes  in  program  requirements,  either  with  or 
without  additional  funding 

Affordability 

Decision  by  OSD,  the  service,  or  Congress  to  change  the 
program  because  of  changed  priorities  (reprogramming 
decisions) 

Quantity 

Schedule  changes 

Increase  or  decrease  in  the  quantity  of  systems  built 

Decision  by  OSD,  Congress,  or  the  service  to  change  the 
program  schedule  (extend,  contract,  or  restructure) 

Inter-  or  intra¬ 
program  transfers 

Transfers  of  planned  funding  within  a  program  (between 
development  and  procurement  or  O&M)  or  between 
programs 

Financial 

Exchange  rate 

Program  cost  changes  associated  with  differences 
between  predicted  and  actual  exchange  rates 

Inflation 

Program  cost  changes  associated  with  differences 
between  predicted  and  actual  inflation 

Miscellaneous 

Error  corrections 

Variances  from  errors  in  the  SARs 

Unidentified 

Unexplained  variances 

External  events 

External  event  affecting  program  cost,  schedule,  or 
technology 

SOURCE:  Bolten  et  al„  2008. 

on  this  methodology  can  be  difficult  and  somewhat  subjective.  As  a 
result,  the  reader  should  view  any  single  value  for  a  category — Errors, 
Decisions,  and  so  forth — as  having  significant  uncertainty,  and  values 
in  the  subcategories  should  be  viewed  as  having  even  larger  uncertainty 
in  relative  terms. 
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Errors 

The  Errors  category  is  composed  of  three  subcategories:  cost  estimates, 
schedule  estimates,  and  technical  issues.  Variances  that  fall  into  the 
cost  estimation  subcategory  are  revised  cost  estimates  that  are  associ¬ 
ated  with  meeting  the  required  capabilities  as  defined  at  the  program’s 
MS  B.  Variances  that  fall  into  the  schedule  estimation  subcategory  are 
those  causing  changes  to  dates  for  any  program  milestone  occurring 
after  MS  B.  Estimate  revisions  of  costs  or  schedule  within  the  Errors 
category  may  or  may  not  involve  increased  work  scope,  but  they  do 
not  include  revised  estimates  resulting  from  changes  in  the  system’s 
requirements. 

Variances  that  fall  into  the  technical  issues  subcategory  are  those 
associated  with  developing,  qualifying,  and  fielding  the  hardware  sys¬ 
tems,  subsystems,  and  software  that  provide  the  capability  specified  at 
MS  B.  Difficulties  with  developing  these  and  any  replacement  hard¬ 
ware  or  software  (should  the  former  be  abandoned  or  found  inade¬ 
quate)  fall  into  this  subcategory.  Difficulties  in  developing  technologies 
that  provide  capability  beyond  that  specified  at  MS  B  are  not  included 
in  the  technical  issues  subcategory. 

The  descriptions  of  cost  variances  in  the  SARs  vary  in  quality 
from  program  to  program  and  within  any  single  program  over  time. 
Some  are  quite  specific  in  describing  causal  relationships  resulting  in 
cost  growth,  while  others  are  brief  and  ambiguous.  Reading  the  gen¬ 
eral  narrative  provided  in  each  SAR  provides  an  overview  of  the  pro¬ 
gram’s  status  and  concerns  at  that  point  in  time.  Reading  the  cost  vari¬ 
ance  explanations  in  this  context  strongly  suggests  that  many  cost  and 
schedule  estimating  errors  are  the  result  of  technical  issues  but  are  not 
often  explicitly  identified  as  such  in  the  variance  explanation.  This  per¬ 
ception  was  confirmed  in  our  expert  interviews  with  individuals  associ¬ 
ated  with  the  SBIRS-High  and  GPS  programs.  As  a  result,  we  believe 
that  technical  issues  almost  certainly  played  a  larger  part  in  cost  growth 
than  we  are  able  to  demonstrate  conclusively  using  our  SAR  categoriza¬ 
tion  methodology.  We  believe  that  cost  growth  attributed  to  the  cost 
and  schedule  subcategories  are  overstated,  while  that  attributed  to  the 
technical  issues  subcategory  is  understated.  Fortunately,  all  of  these 
variances  are  captured  within  the  same  overall  Errors  category. 
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Decisions 

The  Decisions  category  includes  program  changes  that  are  within 
the  control  of  an  entity  of  authority — the  program  office,  SMC,  the 
Air  Force,  OSD,  Congress,  or  the  President.  Decisions  is  composed 
of  five  subcategories:  requirements,  affordability,  quantity,  schedule, 
and  inter-  and  intra-program  transfers.  The  requirements  subcategory 
includes  cost  growth  associated  with  increases  and  decreases  in  desired 
capability  of  the  system.  Also  included  are  all  subsequent  cost  esti¬ 
mate,  schedule  estimate,  and  technology-related  cost  increases  asso¬ 
ciated  with  requirements  changes.  Note  that  these  same  subcatego¬ 
ries  are  found  under  the  Errors  category,  but  such  variances — those 
that  are  associated  with  requirements  added  to  the  program  after  its 
MS  B — are  considered  cost  growth  associated  with  the  decision  to 
change  the  system’s  desired  capability. 

Requirements  are  occasionally  scaled  back  without  the  report¬ 
ing  of  an  associated  cost  avoidance  (or  savings).  This  typically  occurs 
at  the  time  of  a  program  re-baseline  during  the  major  development 
phase,  when  some  maturing  system  components  are  not  providing 
the  required  capability.  If  the  anticipated  performance  is  close  to  what 
was  originally  specified,  a  decision  is  often  made  to  relax  slightly  the 
original  requirement  rather  than  incur  significant  additional  cost  and 
schedule  slippage  to  achieve  the  original  specification.  The  effect  of 
relaxing  requirements  without  reporting  associated  cost  avoidance  is 
the  underestimation  of  the  cost,  either  to  meet  the  relaxed  requirement 
or  to  meet  the  original  requirement.  We  are  not  able  to  correct  for  this 
underestimation. 

The  affordability  subcategory  is  primarily  made  up  of  the  results 
of  reprogramming  decisions  affecting  the  program’s  funding.  These 
reprogramming  actions,  usually  budget  cuts  within  the  Future  Years 
Defense  Program  (FYDP),  are  not  associated  with  requirements 
changes  and  are  beyond  the  program’s  control.  A  higher  authority — 
SMC,  the  Air  Force,  OSD,  Congress,  or  the  President — typically  insti¬ 
tutes  them.  Affordability  variances  tend  to  be  small  but  can  add  up 
over  time.  They  are  usually  accounted  for  as  cost  savings  because  fund¬ 
ing  is  usually  taken  from  a  program.  Some  programs  simply  can  do 
with  fewer  resources  in  the  short  term;  thus,  “nibbling”  at  their  budgets 
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has  no  long-term  effects.  We  believe  that  in  other  programs  most  of 
what  is  “saved”  in  this  subcategory  is  more  than  offset  by  increased  cost 
growth  realized  years  later  that  shows  up  in  other  variance  categories. 
We  have  no  way  of  accounting  for  this  effect. 

The  quantity  subcategory  consists  primarily  of  changes  in  the 
total  number  of  units  acquired.  Cost  growth  associated  with  units 
transferred  between  program  phases — typically,  development  and 
procurement — are  included  in  this  subcategory.  Support  item  quan¬ 
tity  changes  and  other  program  changes  associated  with  a  change  in 
the  number  of  primary  items  (satellites  in  our  two  programs)  are  also 
included  in  the  quantity  subcategory.  While  quantity  changes  can  be 
reasonably  described  as  a  form  of  requirements  change,  we  chose  to 
report  them  in  a  separate  subcategory  under  Decisions. 

The  schedule  subcategory  applies  when  development  milestones  or 
production  rates  are  changed  by  choice,  not  because  of  technical  prob¬ 
lems  and  not  to  free  up  funding  to  apply  against  cost  growth  in  other 
portions  of  the  program.  These  changes  often  take  the  form  of  budget 
cuts  and  are  therefore  production  rate  cuts  in  the  program’s  FYDP 
funding  to  free  up  resources  for  short-term  priorities  external  to  the 
program.  Schedule  change  decisions  with  cost  growth  consequences 
are  at  times  made  in  satellite  programs  because  the  space  vehicles  are 
not  needed  as  quickly  as  previously  thought.  Delaying  these  satellites 
allows  for  incorporation  of  additional  capability,  the  additional  cost  of 
which  is  accounted  for  in  the  requirements  subcategory. 

The  inter-  or  intra-program  transfers  subcategory  applies  when 
work  content  and  its  associated  funding  are  moved  between  program 
phases  or  are  removed  from  or  added  to  the  program.  These  variances 
do  not  represent  “true”  cost  growth  in  either  an  estimating  or  budget¬ 
ing  sense  because  any  change  in  the  program’s  expected  cost  is  not  a 
reflection  on  the  accuracy  of  the  MS  B  cost  estimate.  Funding  and 
associated  work  content  transferred  between  a  program’s  phases  pro¬ 
duce  a  zero-sum  result  for  the  program  as  a  whole;  thus,  there  is  no 
cost  growth  at  that  level.  When  viewed  at  a  higher  level  of  aggrega¬ 
tion,  funding  and  associated  work  content  transferred  to  or  from  the 
program  to  another  also  has  a  zero-sum  cost  effect  because  there  is 
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no  effect  on  the  total  funding  ultimately  needed  to  meet  the  required 
capability.6 

Other  Categories  of  Sources  of  Cost  Variance 

The  RAND  cost  variance  methodology  identifies  two  additional  vari¬ 
ance  categories:  financial  changes  and  miscellaneous.  These  variance 
categories  are  viewed  as  being  beyond  the  control  of  the  cost  estima¬ 
tion  community  and  program  management,  and  they  are  external  to 
the  program  government  entities.  They  do  not  originate  from  either  a 
mistake  or  a  decision. 

The  Financial  category  applies  to  cost  growth  variances  result¬ 
ing  from  two  sources:  differences  between  expected  inflation  rates  and 
their  actual  values,  and  variances  resulting  from  changes  in  exchange 
rates.  This  latter  variance  source  only  applies  to  systems  with  signifi¬ 
cant  foreign  content,  and  thus  does  not  apply  to  our  two  case  study 
space  systems. 

The  Miscellaneous  category  includes  SAR  error  corrections, 
unidentified  variances,  and  external  events.  The  external  events  sub¬ 
category  applies  to  cost  variances  resulting  from  an  event  beyond  the 
control  of  stakeholders.  Examples  are  ship  program  cost  growth  result¬ 
ing  from  the  devastation  of  a  hurricane  or  other  natural  phenomena, 
launch  vehicle  program  cost  growth  resulting  from  shifting  of  payload 
types  to  various  launch  vehicle  programs  in  the  aftermath  of  the  loss  of 
the  space  shuttle  Challenger,  and  aircraft  program  cost  growth  result¬ 
ing  from  protracted  strikes  at  key  contractor  locations.  This  category 
is  seldom  used  and  is  not  used  in  either  of  the  space  system  programs 
examined  here. 

The  unidentified  costs  subcategory  applies  only  when  a  cost  vari¬ 
ance  is  of  unknown  origin.  This  occurs  when  cost  estimates  and  vari¬ 
ances  in  successive  SAR  reports  are  not  mathematically  connected  (no 
audit  trail),  and  when  programs  are  divided  with  no  guidance  regard¬ 
ing  which  prior  variances  belong  to  each  piece  of  the  program.  The 
error  corrections  subcategory  applies  when  variances  are  described  as 


6  Note  that  in  the  RAND  Cost  Growth  Database  intra-program  transfers  are  specifically 
excluded  from  the  cost  growth  factor  calculations  for  the  reasons  cited  above. 
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corrections  from  “errors”  in  prior  reports  with  no  further  information 
given.  This  last  subcategory  is  seldom  used,  and  is  not  used  in  either  of 
our  space  system  programs. 

Underestimation  of  Errors;  Overestimation  of  Decisions 

At  one  or  more  points  after  MS  B,  programs  typically  re-baseline  their 
plan  for  the  remainder  of  the  system’s  acquisition  phase.  A  new  plan 
is  developed  for  the  remaining  execution  of  development  and  produc¬ 
tion  that  includes  updated  cost,  schedule,  and  performance  specifica¬ 
tions  providing  the  desired  capabilities.  If  the  program’s  requirements 
have  not  changed  substantially  since  their  specification  at  MS  B,  then 
this  re-baselining  does  not  create  a  problem  with  our  cost-variance  cat¬ 
egorization  methodology  because  the  additional  costs  can  be  clearly 
assigned  to  the  appropriate  Errors  or  Decisions  category,  depending 
upon  the  reasons  behind  the  re-baselining.  However,  if  requirements 
have  changed  substantially  before  the  re-baseline  or  as  part  of  it,  then 
the  variances  associated  with  the  re-baseline  are  at  least  partially,  but 
may  not  be  wholly,  a  result  of  the  new  requirements.  This  is  where  a 
problem  occurs. 

Some  contextual  information  must  be  kept  in  mind  to  get  a  better 
understanding  of  the  problem.  The  new  program  execution  plan  that  is 
associated  with  an  updated  program  baseline  necessarily  includes  cost 
estimates  and  milestone  schedules  covering  the  remainder  of  the  sys¬ 
tem’s  acquisition  phase.  If  requirements  exceed  those  defined  at  MS  B, 
then  the  planned  system  design  meets  the  original  requirements  plus 
those  added  since  that  time.  Inherent  in  such  a  plan,  and  an  imbed¬ 
ded  subset  thereof,  are  updated  plans  to  provide  the  system  capability 
defined  at  MS  B.  Any  difficulties  associated  with  that  effort  that  were 
not  previously  acknowledged  and  accounted  for  as  cost  variances  are  now 
incorporated  into  the  new  baseline.  When  difficulties  have  not  been 
disclosed  before  the  re-baseline,  they  are  almost  never  specifically  dis¬ 
closed  at  the  time  of  the  re-baseline.  As  a  result,  their  associated  cost 
growth  is  not  separately  identified  within  the  re-baselined  program 
plan. 

The  practice  of  folding  the  plan’s  higher  estimates  into  a  re¬ 
baselined  program  to  meet  the  requirements  defined  at  MS  B,  as  well 
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as  those  defined  since  then,  is  referred  to  in  the  acquisition  commu¬ 
nity  as  “getting  well.”  It  is  a  common  practice  because  few  MDAPs 
have  stagnant  system  requirements  over  their  considerable  acquisition 
time  lines  and  because  SARs  are  congressionally  mandated  “oversight” 
documents  required  by  that  very  entity  with  the  power  to  enhance, 
scale  back,  or  cancel  the  program’s  funding.  The  former  point  pro¬ 
vides  ample  opportunity  for  the  “get  well”  scenario  to  present  itself;  the 
latter  provides  strong  incentives  for  the  program  leadership  to  take  best 
advantage  of  the  added  requirements  to  fold  into  the  new  plan  all  the 
“fixes”  it  knew  would  be  required  but  was  slow  to  acknowledge  fully 
internally  and  report  externally. 

The  result  for  our  cost  variance  allocation  scheme  is  that,  within 
the  variance(s)  reported  as  part  of  a  program  re-baseline,  we  cannot 
break  out  that  portion  of  cost  growth  and  schedule  slips  associated 
with  the  original  system  capabilities  from  that  portion  associated  with 
the  added  requirements.  Absent  this  insight,  we  are  obligated  to  allo¬ 
cate  all  cost  variance  recognized  at  the  time  of  a  program  re-baseline 
to  the  added  requirements,  that  is,  the  requirements  category  of  Deci¬ 
sions.  This  has  the  effect  of  underreporting  cost  growth  in  the  Errors 
category  and  overreporting  it  in  the  Decisions  category. 

The  implications  for  our  two  space  programs  are  as  follows: 
SBIRS-High  has  had  relatively  little  change  in  requirements;  thus,  the 
issue  most  likely  does  not  apply.  GPS  has  had  tremendous  require¬ 
ments  growth;  thus,  the  issue  almost  certainly  has  caused  underesti¬ 
mation  of  cost  growth  attributed  to  the  Errors  category,  along  with  a 
corresponding  overestimation  of  cost  growth  attributed  to  the  Deci¬ 
sions  category. 

The  RAND  cost  variance  methodology  strives  to  identify  the 
main  causes  of  cost  growth.  The  methodology  is  far  more  useful  than 
that  used  in  the  SARs,  but  it  suffers  somewhat  from  insufficient  con¬ 
textual  information  provided  by  the  SARs.  More-detailed  variance 
background  information  would  enhance  the  accuracy  of  assigning 
variances  to  the  causal  categories.  In  the  absence  of  that  information, 
the  methodology  provides  a  reasonable  gauge  for  the  relative  propor¬ 
tions  of  cost  growth  attributable  to  errors,  decisions,  and  other  causes 
in  any  MDAP. 
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We  now  turn  to  a  chronological  description  of  cost  growth  in  the 
SBIRS-High  and  GPS,  citing  the  major  events  and  potential  causes 
over  ten  years  in  each  program. 


Categorization  and  Magnitude  of  SBIRS-High  Cost 
Variance 

Figure  2.1  shows  estimates  for  SBIRS-High  program  segments  at  vari¬ 
ous  points  in  the  past — year-end  in  all  cases  except  the  point  of  origin 
on  the  far  left.  These  estimates  were  taken  from  the  program’s  annual 
SARs.  The  lack  of  data  for  2000  resulted  from  the  absence  of  a  SAR  at 
the  end  of  that  year.  Four  estimates  are  tracked.  The  aggregate  cost  for 

Figure  2.1 

SBIRS-High  Program  Cost  Estimates  over  Time 


NOTES:  PM  EAC  =  Program  manager  estimate  at  completion;  LMSSC  =  Lockheed 
Martin  Space  Systems  Company. 
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developing  and  procuring  two  highly  elliptical  earth  orbit  (HEO)  pay- 
loads,  five  (or  fewer)  geosynchronous  earth  orbit  (GEO)  satellites,  and 
the  ground  segment  controlling  the  constellation  are  shown  in  the  top 
line.7  The  line  just  below  it  contains  estimates  for  only  the  development 
portion  of  the  program.  This  funding  is  a  subset  of  the  prior  one.  The 
bottom  two  lines  show  the  program  manager’s  and  prime  contractor’s 
estimates  at  completion  (EACs)  for  the  engineering  and  manufactur¬ 
ing  development  (EMD)  contract  for  the  program.8  The  lines  overlap 
from  the  MS  B  date  through  1999.  At  some  points  thereafter,  the  pro¬ 
gram  manager’s  EAC  is  higher  than  that  of  the  contractor. 

The  SBIRS-High  program  enjoyed  remarkably  stable  require¬ 
ments  from  1996  through  2005  but  encountered  much  difficulty  keep¬ 
ing  to  its  planned  budget  and  schedule.  The  result  was  that  an  inordi¬ 
nate  number  of  new  Acquisition  Program  Baselines  (APBs)  and  more 
than  200  cost  variances  were  identified  in  the  program’s  SARs  over 
the  ten  years.  The  chronology  of  APBs,  selected  cost  variances,  and 
key  contextual  information  about  the  program’s  progress  are  described 
below. 

The  program’s  original  MS  B  baseline  was  established  on  October 
3,  1996.  The  associated  cost  estimate  was  part  of  a  Defense  Acqui¬ 
sition  Executive  (DAE)  APB  for  the  complete  acquisition  program. 
The  program  consisted  of  five  GEO  satellites,  two  HEO  payloads,  a 
ground  control  segment,  and  software  that  facilitated  functioning  of 
the  SBIRS  system  and  its  integration  with  the  existing  Defense  Sup¬ 
port  Program  (DSP)  satellites  and  their  system  software.  The  program 
was  to  be  funded  almost  entirely  with  development  money;  it  con¬ 
tained  no  procurement  funding.  The  acquisition  program  estimate  was 
$3.66  billion  and  included  annual  funding  through  fiscal  year  (FY)06. 
The  initial  value  of  the  EMD  contract  and  its  EAC  was  just  $1,905  bil- 


7  Cost  growth  in  the  program’s  acquisition-related  O&M  was  substantial,  and  growth  in 
MilCon  was  also  significant.  However,  we  chose  to  concentrate  our  analysis  on  the  develop¬ 
ment  and  procurement  portions  of  the  program  because,  in  aggregate,  they  account  for  more 
than  90  percent  of  program  funding. 

8  Note  that  the  EMD  effort  is  a  subset  of  the  total  development  effort.  The  latter  also 
includes  developmental  activities  occurring  prior  to  the  program’s  EMD  phase. 
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lion,  which  was  far  less  than  the  total  program  estimate.9  These  values 
are  above  the  “MS  B”  at  the  left-hand  side  of  the  x-axis  of  Figure  2.1. 

Shortly  after  the  EMD  contract  was  awarded,  its  scope  was  signif¬ 
icantly  reduced,  but  the  EMD  contract  value  was  left  unchanged.  The 
program  content  reduction  in  scope  removed  the  final  two  GEO  sat¬ 
ellites  from  EMD  and  put  them  into  a  newly  created  production  pro¬ 
gram.  The  absence  of  an  EAC  cut  associated  with  the  EMD  program 
content  reduction  in  scope  implies  that  the  program  office  expected 
that  the  effort  remaining  in  the  EMD  contract — the  designing,  build¬ 
ing,  testing,  and  delivering  the  two  HEO  payloads  and  first  three  (no 
longer  five)  GEO  satellites — would  be  more  costly  than  the  contract 
value  at  MS  B.  We  believe  the  program  office’s  estimate  at  the  time 
of  MS  B  anticipated  that  the  contractor  would  have  difficulty  keeping 
within  the  contract  value  and  reflected  this  in  a  program  baseline  esti¬ 
mate  that  was  far  higher  than  that  of  the  program’s  primary  contract. 

The  program  adopted  its  second  APB  estimate  about  one  and  a 
half  years  after  the  MS  B,  on  March  19,  1998.  Cost  estimates  associ¬ 
ated  with  this  ABP  are  similar  to  those  shown  above  “1998”  on  the 
x-axis  of  Figure  2.1.  The  new  DAE-approved  APB  officially  recognized 
the  moving  of  the  final  two  GEO  satellites,  GEOs  4  and  5,  from  the 
EMD  program  into  the  new  procurement  program.  The  total  SBIRS 
program  was  now  expected  to  cost  $4.15  billion  through  FY06.  Note 
that  the  top  line  in  Figure  2.1  shows  a  value  of  less  than  $4  billion 
in  1998.  This  value  represents  development  and  procurement  efforts 
only,  with  the  difference  accounted  for  in  acquisition  related  O&M 
and  MilCon  funding. 

As  Figure  2.1  shows,  cost  growth  recognized  by  the  program  was 
relatively  low  in  its  first  three  years  after  MS  B.  The  program’s  total 
expected  cost  increased  about  11  percent,  or  about  $400  million.  This 
is  a  modest  amount  compared  with  other  MDAPs  at  this  point  in  the 
acquisition  process.  A  number  of  changes  in  the  program  over  that 
period  contributed  to  both  cost  estimate  increases  and  decreases. 


9  The  major  contracts  in  an  MDAP  typically  account  for  85  to  90  percent  of  the  program’s 
total  funding.  In  estimates  at  the  MS  B  for  SBIRS,  only  52  percent  was  accounted  for  in  the 
primary  contract. 
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The  first  was  the  moving  of  the  final  two  GEO  satellites  from 
the  EMD  phase  to  the  newly  created  procurement  phase  in  November 
1996.  One  month  later,  in  December  1996,  a  cost  increase  of  $188 
million  was  recognized  and  associated  with  the  creation  of  the  procure¬ 
ment  program.10  Based  upon  the  perceived  progress  made  by  the  con¬ 
tractor  as  indicated  at  numerous  design  milestones  for  both  the  space 
and  ground  segments,  the  program  lowered  its  cost  estimate  for  the 
five  GEO  satellites  by  $285  million  in  the  December  1997  SAR.  This 
was  done  in  spite  of  the  recognition  of  a  four-month  delay  in  both  the 
space  and  ground  segment  software  development  and  the  delivery  of  all 
GEO  satellites,  and  a  three-month  delay  in  HEO  sensor  delivery. 

During  1998,  the  program  recognized  a  two-year  or  longer  delay 
in  the  planned  date  for  the  first  and  all  subsequent  GEO  launches, 
eight-month  slips  in  each  HEO  sensor  delivery,  a  33-month  slip  in  the 
“Increment  2”  capability,* 11  and  a  27-month  slip  in  initial  operational 
capability  (IOC).  These  schedule  slips  resulted  from  the  longer  life 
expectancy  of  the  existing  DSP  satellites,  the  desire  to  synchronize  the 
program  with  the  new  schedule  for  the  National  and  Theater  Missile 
Defense  program,  and  the  desire  to  free  up  FYOO  funds.  These  schedule 
slips  added  $494  million  to  the  total  expected  cost. 

Around  the  same  time,  the  third  GEO  satellite  was  moved  from 
the  EMD  program  to  the  procurement  program,  and  funding  for  the 
fourth  and  fifth  GEOs  in  the  procurement  program  was  cut,  with  a  net 
funding  change  of  $90  million  removed  from  the  program.  The  reduc¬ 
tion  in  EMD  contract  scope — removal  of  the  third  GEO — was  carried 
out  without  adjusting  the  development  contract  value  (EAC),  again 
indicating  cost  growth  in  the  remaining  work  content  of  the  EMD 
contract.  Block  II  requirements  providing  additional  capability  to  the 
third  through  fifth  GEO  satellites,  along  with  $215  million  in  fund¬ 
ing,  were  also  added  in  1998.  These  decisions,  along  with  a  number  of 
smaller  program  cost  variances  and  additional  schedule  slips  of  two 


10  The  linking  of  the  $188  million  with  the  establishment  of  a  production  phase  is  not  stated 
in  the  SAR — this  information  came  from  our  interviews. 

11  The  program  is  broken  up  into  Increment  1  and  Increment  2.  The  latter  essentially  repre¬ 
sents  the  functioning  of  the  fully  operational  SBIRS  system. 
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and  six  months  in  the  Increment  1  software  development  and  testing 
and  the  first  HEO  sensor  delivery  date,  respectively,  resulted  in  an  APB 
with  a  total  program  estimate  of  $4.15  billion. 

After  another  18  months,  the  DAE  approved  the  program’s  third 
APB  on  August  13,  1999.12  The  primary  purpose  of  this  APB  was  to 
recognize  the  prior  two-year  stretch-out  of  the  program,  with  funding 
now  planned  through  FY08.  The  total  program  cost  estimate  remained 
$4.15  billion.  Major  cost  variances  leading  up  to  the  new  baseline 
during  1999  included  purchase  of  GEO  satellites  three  through  five 
as  a  block  with  associated  savings  of  $155  million,  an  additional  $190 
million  increase  from  prior  estimates  for  the  program  restructure,  and 
revised  inflation  indices  saving  the  program  $145  million. 

Cumulative  SBIRS-High  Program  Cost  Variances 

As  noted  earlier,  over  the  ten-year  period  some  200  variances  were 
reported  in  the  SBIRS-High  program  SARs.  The  total  included  81  in 
development;  60  in  procurement;  18  in  MilCon;  and  41  in  acquisition- 
related  O&M.  Variances  were  reported  in  such  a  way  that  they  could 
not  be  broken  out  among  the  primary  pieces  of  the  program:  GEO 
satellites,  HEO  payloads,  and  the  ground  segments. 

Figure  2.2  presents  the  changes  in  the  development  plus  procure¬ 
ment  program  estimate  over  time  by  variance  category,  showing  the  net 
of  cost  growth  plus  cost  underrun  in  each  subcategory  and  accumula¬ 
tion  over  time.  Nine  subcategories  in  all  apply  to  the  program:  three  in 
the  Errors  category  and  five  in  the  Decisions  category.  The  Inflation/ 
financial  change  category  is  shown  in  light  gray.  Of  these  nine,  five 
caused  substantial  positive  cost  growth,  one  caused  minor  positive  cost 
growth,  one  caused  a  substantial  cost  underrun,  and  two  caused  minor 
cost  underruns. 

The  program  experienced  over  $8  billion  in  positive  cost  vari¬ 
ances,  with  all  but  about  $1  billion  falling  into  the  three  Errors  subcat¬ 
egories  of  cost  estimates,  schedule  estimates,  and  technical  issues.  Just 


12  This  is  counted  from  MS  B;  thus,  this  third  APB  is  the  second  since  the  MS  B  baseline 
was  established. 
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Figure  2.2 

Cumulative  Variances  in  SBIRS-High  Development  and  Procurement  over 
Time  Using  RAND  Methodology 
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under  $0.5  billion  in  cost  growth  was  accounted  for  in  the  decisions- 
related  requirements  subcategory,  and  just  over  $0.5  billion  in  cost 
growth  was  accounted  for  in  the  decisions-related  schedule  subcate¬ 
gory.  The  program  also  had,  on  net,  a  small  cost  growth  in  the  inter-  or 
intra-program  transfers  subcategory  of  the  Decisions  category. 

The  only  subcategory  with  a  substantial  cost  underrun  is  quantity 
within  the  Decisions  category,  at  about  $1.5  billion.  Also  within  the 
Decisions  category  was  a  small  cost  underrun  in  the  affordability  sub¬ 
category.  The  program  experienced  relatively  small  cost  underruns  in 
the  inflation/financial  changes  category. 


SBIRS-High  Cost  Problems  Bottom  Line 

Most  of  the  program’s  cost  growth  resulted  from  errors  in  the  form 
of  inappropriate  cost  and  schedule  estimates  made  by  the  contractor 
and  accepted  by  the  government.  However,  much  underlying  evi¬ 
dence  suggests  that  these  estimating  errors  were  rooted  in  technical 
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problems — both  in  understanding  the  difficulty  of  technological  design 
solutions  required  to  deliver  the  SBIRS  capability  and  in  understand¬ 
ing  the  integration  and  test  efforts  required  to  ensure  that  the  design 
solutions  were  robust.  The  program’s  SARs  were  slow  to  acknowledge 
cost  growth,  waiting  four  years  into  development  to  do  so  in  earnest. 
Specific  technical  issues  contributing  to  cost  growth  were  not  identified 
until  eight  years  after  the  MS  B. 

The  program’s  acquisition  costs  increased  dramatically.  The  total 
program  cost  estimate  grew  from  $3.66  billion  at  MS  B  for  a  program 
including  five  GEO  satellites  to  $10.16  billion  in  March  2006  for  a 
program  with  just  three  of  those  satellites.  If  we  do  some  crude  cal¬ 
culations  to  normalize  for  content — that  is,  adding  back  into  the  pro¬ 
gram  the  expected  cost  of  the  two  deleted  satellites  and  adjusting  for 
requirements  added  to  the  program — the  total  program  estimate  rises 
to  somewhere  between  $10.8  and  $13.1  billion.  This  is  three  to  three 
and  one-half  times  the  MS  B  estimate.13 


Categorization  and  Magnitude  of  Global  Positioning 
System  Cost  Variance  (HR,  IIF,  and  OCS  Segments) 

The  desired  capabilities  of  the  GPS  program  changed  substantially 
between  1996  and  2005.  GPS  Block  IIR-M  and  IIF  satellites  are  the 
latest  generation  of  operational  satellites  to  populate  the  nominal 
24-satellite  constellation  that  provides  highly  accurate,  space-based, 
radio-positioning,  and  navigation  services  for  military  and  commercial 
users.14 


13  The  minimum  number  is  the  March  2006  baseline  estimate  of  $10.16  billion  plus  the 
$1,147  billion  in  funding  removed  from  the  program  when  the  fourth  and  fifth  GEO  satel¬ 
lites  were  removed,  less  the  roughly  $0,471  in  positive  cost  variances  on  net  associated  with 
added  requirements.  The  maximum  number  is  the  March  2006  baseline  estimate  plus  two 
times  (representing  the  value  of  each  the  fourth  and  fifth  GEO  satellites)  the  current  $1,723 
billion  estimate  for  the  planned  single  GEO  satellite  production  program  less  the  same  posi¬ 
tive  cost  variance  for  added  requirements. 

14  This  overview  of  the  early  phases  of  the  GPS  program  is  drawn  largely  from  Scott  Pace, 
Gerald  P.  Frost,  Irving  Lachow,  David  R.  Frelinger,  Donna  Fossum,  Don  Wassem,  and 
Monica  M.  Pinto,  The  Global  Positioning  System:  Assessing  National  Policies ,  Santa  Monica, 
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The  program  originally  began  in  1973  with  12  GPS  I  develop¬ 
mental  satellites  designed  and  manufactured  by  Rockwell  Interna¬ 
tional  (now  Boeing).  The  first  operational  GPS  space  vehicles  (SVs) 
were  Block  II  and  IIAs,  also  developed  by  Rockwell,  first  launched  in 
February  1989.  In  1983  the  Air  Force  awarded  Rockwell  a  contract 
to  build  28  Block  II/IIA  satellites.  Block  IIs  differed  in  weight  and 
shape  compared  with  Block  Is  and  had  superior  security  and  integ¬ 
rity  features.  In  addition,  Block  II  satellites  launched  after  1989,  des¬ 
ignated  Block  IIAs,  had  much  improved  capabilities  for  autonomous 
operation.  The  launch  of  the  24th  Block  II  in  March  1994  completed 
the  GPS  constellation,  which  consisted  of  24  Block  II/IIAs  and  one 
Block  I.  Rockwell  built  four  additional  Block  IIAs  as  spares.  In  1995, 
the  Air  Force  announced  that  the  GPS  constellation  had  achieved  full 
operational  capability  (FOC).  The  last  Block  IIA  was  placed  in  orbit  in 
November  1997. 

In  June  1989,  the  Air  Force  awarded  a  contract  for  20  GPS  Block 
IIRs  with  an  option  for  six  more,  the  next-generation  replacement  space 
vehicles  for  the  II/IIAs.  The  Block  HR  was  developed  by  General  Elec¬ 
tric  (now  Lockheed  Martin)  after  an  open  competition.  The  first  HR 
was  launched  in  1997.  Block  IIRs  are  modestly  more  capable  satellites 
intended  to  replace  Block  II/IIAs  to  maintain  a  nominal  24-satellite 
constellation. 

In  June  1995,  the  Air  Force  released  a  draft  request  for  proposals 
to  industry  for  33  Block  IIF  follow-on  satellites  to  replace  the  IIAs  and 
IIRs.  Like  earlier  generations  of  GPS  SVs,  the  Block  IIF  was  originally 
envisioned  as  broadly  meeting  the  same  high-level  operational  require¬ 
ments  but  with  important,  though  relatively  modest,  performance 
improvements.  Boeing  won  the  contract  for  up  to  33  Block  IIFs  in 
April  1996  after  an  open  competition.  Initially  the  Air  Force  awarded 
Boeing  a  procurement  contract  for  6  Block  IIFs  with  options  for  the 
remaining  27. 

However,  the  GPS  modernization  program  went  through  a  major 
restructuring  in  2000.  The  IIF  modernization  effort  that  began  in 


Calif.:  RAND  Corporation,  MR-614-OSTP,  1995,  pp.  238-247,  supplemented  by  informa¬ 
tion  from  the  GPS  JPO. 
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1996  became  much  more  complex  by  January  1999  with  Vice  Presi¬ 
dent  Gore’s  announcement  of  two  additional  civilian  signals  for  GPS. 
In  June  1999,  a  new  operational  requirements  document  (ORD)  and 
new  key  performance  parameters  (KPP),  including  jam  resistance  from 
space,  backward  compatibility,  and  system-level  time  transfer,  were 
added  to  the  desired  system  capabilities. 

Key  to  the  adoption  of  new  capabilities  for  satellites  not  yet 
launched  was  the  longer  life  expectancy  of  GPS  satellites  already  in 
orbit.  As  of  the  end  of  1999,  almost  the  entire  operational  GPS  con¬ 
stellation  consisted  of  Block  II  and  IIA  satellites.  During  1997,  the  life 
expectancy  of  the  Block  II  satellites  increased  from  an  estimated  6.5 
years  to  8.5  years.  Between  1996  and  1999  that  for  the  Block  IIA  satel¬ 
lites  increased  from  6.5  years  to  10.6  years.  These  longer  life  expectan¬ 
cies  meant  that  replacement  satellites  would  be  needed  years  later  than 
originally  anticipated.  This  created  the  opportunity  to  upgrade  replace¬ 
ments  before  their  launch. 

These  circumstances  led  to  the  late  1999  Defense  Resources 
Board  (DRB)  approval  for  modification  of  up  to  12  HR  satellites  with 
a  second  civilian  signal  and  full-earth  coverage  of  the  military  signal. 
These  upgraded  HR  satellites  were  designated  IIR-Ms.  Also  approved 
was  the  modification  of  the  first  six  IIF  satellites  with  a  second  and 
third  civilian  signal  and  full-earth  coverage  of  the  military  signal.  Con¬ 
gress  approved  the  modernization  plan  in  August  2000. 

In  addition  to  the  capability  enhancements,  a  large  fraction  of 
the  envisioned  program  was  transferred  into  a  future  GPS  system.  This 
shift  is  discussed  in  more  detail  below. 

These  changes  considerably  confound  our  ability  to  measure  and 
categorize  GPS  program  cost  variances,  and  thus  our  categorization  of 
cost  variances  is  not  as  precise  as  we  would  like.  Contrary  to  percep¬ 
tions  commonly  found  in  the  media  and  within  the  defense  acquisition 
community,  we  believe  that  the  program’s  cost  growth  and  schedule  slip¬ 
page  during  this  period  are  considerable — when  adjusted  for  content — 
and  that  the  program  is  not  quite  as  good  a  model  for  an  efficient  and 
effective  acquisition  of  a  space  system  as  is  sometimes  thought. 

Our  analysis  of  the  GPS  program  excludes  the  major  program 
segments  that  occurred  before  1996.  Specifically,  we  exclude  the  initial 
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development  effort  for  the  system  in  the  1970s  and  1980s  that  included 
the  12  Block  1  satellites  and  the  initial  constellation  of  production  sat¬ 
ellites — the  28  Block  II  and  IIA  satellites.  We  also  exclude  those  pro¬ 
gram  segments  that  represent  the  far-term  future  of  the  program — that 
is,  the  Operational  Control  System  (OCX)  ground  segment  capability 
and  the  Block  III  satellites. 

We  measure  cost  growth  from  a  baseline  of  the  May  3,  1996,  Air 
Force  Acquisition  Executive  APB,  which  had  a  value  of  $8.98  billion 
for  78  satellites.15  At  that  time,  the  first  efforts  to  modernize  the  exist¬ 
ing  GPS  constellation  occurred.  The  modernization  OCS  contract  and 
the  design  and  production  contract  for  the  first  six  follow-on  (IIF)  sat¬ 
ellites  were  awarded  in  April  1996.  These  contracts  were  awarded  to 
Boeing,  which  was  not  the  contractor  that  designed  and  built  the  HR 
satellites  in  production  at  the  time. 

The  Boeing  contract  was  awarded  at  a  much  lower  value  than 
expected,  and  as  a  result  $580  million  was  removed  from  the  program 
estimate  by  the  end  of  1996.  This  “savings”  was  short-lived  because 
recognition  of  underestimation  of  costs  for  current  and  future  work  in 
each  of  the  years  1997,  1998,  and  1999  resulted  in  $860  million  added 
to  program  estimates.  The  multiyear  procurement  strategy  for  FY99-04 
satellite  buys  was  abandoned  in  1998;  the  same  strategy  for  FY05-08 
was  likewise  abandoned  in  1999.  The  combined  effect  of  these  acquisi¬ 
tion  strategy  changes  was  an  increase  of  $230  million.  During  those 
two  years,  the  FY00  satellite  buy  was  cancelled,  as  were  the  FY01  and 
02  satellite  buys.  The  cancellation  of  these  nine  satellites,  made  pos¬ 
sible  by  the  longer-than-expected  life  of  the  satellites  in  orbit,  saved  the 
program  an  estimated  $250  million.  Also,  in  the  late  1990s,  estimates 
for  future  inflation  declined  markedly,  reducing  the  program  estimate 
by  $400  million.  The  combined  effect  of  these  cost  variances  was  a  net 
decrease  of  $140  million  in  the  aggregate  estimate  for  development  and 
production  of  GPS. 

The  modernization  effort  that  begun  in  1996  became  much  more 
involved  by  January  1999  with  Vice  President  Gore’s  announcement 


15  This  baseline  estimate  and  all  subsequent  baseline  estimates  exclude  all  acquisition  costs 
for  the  40  Block  I/II/IIA  satellites. 
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of  two  additional  civilian  signals  for  GPS,  as  described  above,  which 
added  costs  estimated  at  $426  million.16  Cost  growth  associated  with 
the  new  capabilities  amounted  to  $1.56  billion  by  the  end  of  1999.  As 
a  direct  result  of  all  the  changes,  the  existing  IIF  program  was  put  on 
hold  in  October  1999. 

Before  the  planned  upgrades,  the  overall  capabilities  of  the  IIR 
satellites  built  by  Lockheed  Martin  and  those  of  the  IIF  satellites  built 
by  Boeing  were  intended  to  be  approximately  the  same.  Also  before  the 
upgrades,  each  contractor  was  responsible  for  developing  the  OCS  func¬ 
tions  for  its  own  satellites,  with  the  intent  that  they  coordinate  efforts 
to  ensure  interoperability.  Our  interviews  revealed  that  this  arrange¬ 
ment  did  not  work  well — i.e.,  the  two  contractors  did  not  effectively 
coordinate  their  continuing  OCS  development  efforts.  When  the  new 
capabilities  were  approved  in  1999,  the  program  office  saw  the  need 
to  change  the  contractual  arrangement  between  the  two  contractors 
to  ensure  an  integrated  OCS  for  all  GPS  satellites.  In  October  1999, 
the  Lockheed  Martin  OCS  efforts  were  consolidated  under  the  Boeing 
modernization  OCS  contract,  with  Boeing  taking  the  overall  lead. 

As  of  the  end  of  1999,  the  IIR  satellite  contract  with  Lockheed 
Martin  called  for  21  satellites,  18  of  which  were  already  delivered.  Three 
of  those  18  had  been  launched,  but  none  of  the  three  was  fully  opera¬ 
tional.17  This  left  15  completed  and  three  in-production  IIR  satellites 
that  had  not  yet  been  launched  and  were  therefore  potentially  available 
for  upgrading  with  the  added  capabilities.  The  upgrade  contract  for  the 
IIR  satellites  called  for  the  upgrading  of  as  many  as  12  of  these  satel¬ 
lites.  The  implication  was  that  the  remaining  six  IIR  satellites  would 
be  required  to  be  in  orbit  before  the  upgrade  on  any  IIR  was  complete. 
The  upgraded  satellite  configuration  was  designated  IIR-M. 

None  of  the  six  IIF  satellites  originally  placed  on  contract  with 
Boeing  in  1996  was  delivered  by  the  time  work  was  stopped  in  Octo- 


16  This  estimate  included  $130  million  in  funding  provided  by  the  U.S.  Department  of 
Transportation. 

17  The  first  was  lost  in  a  launch  failure,  the  second  experienced  continuing  on-orbit  test 
problems  but  attained  basic  functionality,  and  the  third,  which  was  recently  launched,  was 
undergoing  on-orbit  testing. 


32  Improving  the  Cost  Estimation  of  Space  Systems 


ber  1999.  Congress  approved  the  modernization  of  all  IIF  satellites; 
thus,  the  six  partially  completed  satellites  and  all  subsequent  build- 
from-scratch  IIF  satellites  required  additional  development  and  manu¬ 
facture  to  meet  the  newly  specified  capabilities.  Upgrades  to  both  the 
Lockheed  Martin  IIR  and  Boeing  IIF  satellites  were  put  on  contract 
just  after  the  August  2000  congressional  approval. 

In  addition  to  the  capability  enhancements,  a  large  fraction  of 
the  envisioned  program  was  transferred  into  a  future  GPS  system.  Its 
satellites  were  designated  GPS  III,  and  their  new  associated  ground 
segment  was  named  OCX.  The  development  funding  for  GPS  III  and 
associated  OCS  functionality  transferred  out  of  the  program  resulted 
in  $680  million  being  removed  from  the  current  program;  the  pro¬ 
curement  funding  for  36  GPS  satellites  and  associated  OCS  opera¬ 
tions  transferred  out  of  the  program  resulted  in  $3,300  million  being 
removed  from  the  current  program.  The  creation  of  the  future  GPS 
III  and  OCX  programs  reduced  the  number  of  new  satellites  yet  to  be 
produced  in  the  current  GPS  program  from  42  to  six. 

In  February  2002,  all  the  above  changes  were  incorporated  into 
a  new  APB.  The  new  APB  was  the  first  time  the  program  baseline  had 
been  updated  since  May  1996.  The  full  modernization  effort  included 
two  additional  civilian  signals,  full-earth  coverage  military  code,  and 
increased  power  in  the  military  signal.  The  APB  approved  by  the  Under 
Secretary  of  the  Air  Force  (USecAF)  was  valued  at  $5.12  billion.  This 
figure  included  the  original  purchase  of  the  21  IIR  satellites,  the  modi¬ 
fication  of  12  of  the  IIR  satellites  with  a  new  civil  signal  and  full  earth 
coverage  for  the  military  signal,  the  original  purchase  of  six  IIF  satel¬ 
lites  and  their  subsequent  modification  with  two  new  civil  signals  and 
full  earth  coverage  for  the  military  signal,  and  the  purchase  of  six  all- 
new  IIF  satellites  with  all  the  capability  of  the  prior  six  modernized  IIF 
satellites.  In  total,  33  satellites  are  in  the  baseline  number:  9  IIR,  12 
IIR-M,  and  12  IIF,  or  36  fewer  satellites  than  the  program  plan  had 
called  for  just  two  years  earlier.18 


18  Sixty-nine  satellites  were  planned  as  of  the  end  of  1999,  a  reduction  of  nine  from  the  May 
1996  baseline. 


Analysis  of  the  Magnitude  and  Sources  of  Cost  Growth  33 


The  2002  re-baseline  was  preceded  by  an  extended  period  of 
requirements  analysis.  In  our  view,  this  added  considerable  technologi¬ 
cal  complexity  and  requirements  to  both  the  space  and  ground  GPS 
segments.  The  new  capabilities  represented  what  amounted  to  a  gen¬ 
erational  update  of  the  system.  The  differences  between  the  old  and 
new  systems  probably  should  have  precipitated  the  declaration  of  a  new 
MDAP  by  the  relevant  Defense  Acquisition  Authority.  Doing  so  would 
have  required  new  MS  B  authority,  and  that  process  would  have  illu¬ 
minated  both  the  costs  and  benefits  of  the  next  generation  of  GPS.  It  is 
not  clear  why  restructuring  as  a  new  program  was  not  pursued. 

Continuing  evolution  of  requirements  and  other  factors  led  to  yet 
another  new  program  baseline  dated  Feb  14,  2003,  representing  the 
third  APB  in  ten  years.  The  USecAF  APB,  which  had  a  program  value 
of  $5.73  billion,  called  for  37  satellites  in  total,  with  the  modernization 
of  eight  existing  HR  satellites,  rather  than  12.  This  baseline  included 
four  additional  newly  built  satellites  with  an  associated  $288  million 
for  their  production,  and  mandated  that  a  “Flexible  Power”  capability 
be  added  to  all  IIR-M  and  IIF  satellites,  with  an  associated  $300  mil¬ 
lion  added  to  the  program. 

Over  the  next  three  years,  program  costs  continued  to  grow.  Some 
$200  million  was  added  for  IIF  satellite  and  OCS  cost  growth  result¬ 
ing  from  ground  system  manufacturing  difficulties.  In  2004,  three 
satellites  and  $237  million  were  added  to  the  current  program  plan 
because  of  schedule  slips  in  developing  the  future  GPS  III  program, 
and  increases  in  cost  estimates  for  OCS  resulted  in  an  additional  $210 
million.  In  2005,  more  functionality  was  transferred  out  of  the  pro¬ 
gram  to  the  future  GPS  III  and  OCX,  removing  $190  million  from  the 
current  program  and  adding  years  of  additional  OCS  support  for  the 
IIF  program  at  a  cost  of  $170  million. 

Figure  2.3  shows  the  total  GPS  IIR,  IIR-M,  IIF,  and  OCS  pro¬ 
gram  values  over  time.  The  break  in  the  year  2000  resulted  from  the 
absence  of  a  SAR  at  the  end  of  that  year.  Note  that  the  total  pro¬ 
gram  value  (as  shown  in  the  blue  line)  is  substantially  less  after  2000 
than  it  was  before  2000,  and  is  about  17  percent  lower  in  2005  than  it 
was  in  1995.  This  decline  resulted  from  the  removal  of  about  half  the 
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Figure  2.3 

A  Ten-Year  Look  at  SAR  Cost  Estimates  for  GPS  Development  and 
Procurement 
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total  planned  satellites  from  the  program.  Adjusted  for  satellite  quan¬ 
tity  change  (orange  line),  the  program’s  total  value  increased  by 
$2.6  billion,  or  about  34  percent,  from  1996  through  2005. 19  The  value 
of  program  expenditures  for  development  increased  by  about  a  third 
over  the  ten  years,  while  that  for  procurement  decreased  by  about  a 
third. 

Over  the  ten-year  time  frame,  GPS  program  SARs  reported  some 
169  cost  variances.  The  total  included  67  in  development  and  102  in 
procurement.  The  acquisition  program  does  not  include  funding  from 


19  The  quantity  adjustment  was  made  using  the  average  unit  procurement  cost  for  those 
units  remaining  in  the  program  in  each  year’s  estimate  for  the  entire  production  run.  The 
per-satellite  adjustment  value  is  unique  to  each  annual  estimate,  steadily  increasing  from 
about  $95  million  in  1999  to  about  $105  million  in  2005. 
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O&M  or  MilCon  accounts.  We  were  able  to  allocate  most  of  the  vari¬ 
ances  to  their  applicable  program  segment — IIR-M  satellites,  IIF  satel¬ 
lites,  and  OCS  ground  segment — in  the  development  program. 

The  cumulative  development  variances  over  time  for  each  pro¬ 
gram  segment  are  shown  in  Figure  2.4.  Those  directly  attributed  to  the 
IIR  satellites  (red  line)  are  quite  small  when  combined  with  the  later 
modernization  of  eight  of  them.  The  cumulative  development  vari¬ 
ance  for  the  IIF  satellites  (orange  line)  is  more  than  $800  million,  and 
that  for  OCS  development  (gray  line)  is  almost  $600  million.  These 
increases  are  somewhat  offset  by  two  areas  of  decreases.  Almost  $200 
million  in  cumulative  development  cost  underrun  was  not  identified 
with  specific  program  segments  (blue  line),  and  almost  $700  million  in 
future  development  efforts  was  transferred  to  the  GPS  III  satellites  and 
their  OCX  ground  segment  (purple  line). 

Figure  2.4 

Cumulative  Development  Cost  Variance,  by  Program  Segment  over  Time 
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The  time  trends  in  the  figure  suggest  that  beginning  in  1999, 
when  the  costs  of  added  capabilities  began  to  come  into  better  focus, 
a  decision  to  shift  many  of  these  capabilities  to  the  following  genera¬ 
tion’s  system  was  made.  Had  this  not  been  done,  the  development  cost 
growth  in  the  program  would  have  been  at  least  the  currently  measured 
$500  million  plus  the  $700  million  transferred  to  the  future  GPS  space 
and  ground  segments. 

Cumulative  procurement  variances  over  time  for  various  activities 
are  shown  in  Figure  2.5.  We  have  attempted  to  allocate  variances  to 
specific  efforts  in  the  program  and  to  specific  events  that  occurred  over 
the  ten-year  time  frame. 

Figure  2.5 

Cumulative  Procurement  Cost  Variance,  by  Program  Segment  or  Specific 
Effort  over  Time 
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The  modernization  effort  defined  during  1998  and  1999  (green 
line)  added  more  than  $900  million  in  cost  growth  to  the  production 
program.  All  this  growth  occurred  by  the  end  of  1999,  meaning  that 
it  represented  early  estimates  of  the  costs  of  adding  the  new  capabili¬ 
ties  defined  by  the  end  of  that  year  to  future  GPS  satellites.  The  aggre¬ 
gate  cost  growth  represents  an  approximate  $18.8  million  per  unit  cost 
increase  for  each  of  the  48  satellites  remaining  in  the  planned  produc¬ 
tion  program  at  the  time.20 

The  initial  IIF  effort  was  put  on  contract  in  1996  (orange 
line)  with  a  $490  million  reduction  in  the  cost  estimate,  almost 
all  of  which  resulted  from  the  lower-than-expected  contract  value 
awarded  to  Boeing.  The  contract  value  reduction  portion  amounted  to 
$37.5  million  less  for  each  of  the  12  satellites  covered  by  the  contract 
and  its  options.  The  contract  was  signed  for  an  amount  much  lower 
than  that  the  program  office  expected  and  was  almost  certainly  a  result 
of  Total  System  Performance  Responsibility  (TSPR)  and  other  acquisi¬ 
tion  reform  policies  of  the  time.21  Over  the  next  ten  years,  that  reduc¬ 
tion  shrank  to  about  $19.2  million  per  satellite.  Note  that  none  of  the 
12  satellites  has  yet  been  delivered,  and,  according  to  our  program 
office  interviewees,  their  costs  continue  to  grow. 

The  second  piece  of  IIF  cost  growth  (dark  blue  line)  covers  changes 
in  the  configuration  and  some  of  the  changes  in  the  quantity  of  IIF 
satellites  to  be  procured.  The  cost  estimates  for  these  efforts  grew  by 
more  than  $700  million,  ft  appears  that  these  changes  did  not  begin 
until  after  the  modernization  program  was  defined — that  is,  after  the 
February  2002  baseline  was  established.  However,  before  that  date, 
nine  satellites  were  removed  from  the  planned  program.  The  associated 
$246  million  in  cost  savings  was  essentially  offset  by  $241  million  in 
increases  for  acquiring  the  remaining  satellites,  which  was  attributed 
to  changes  in  the  acquisition  strategy  from  multiyear  procurement  to 


20  As  of  December  1999,  the  program  contained  69  satellites.  Of  those,  21 — all  IIR 
models — were  complete  or  nearing  completion  and  were  not  yet  considered  for  upgrading  to 
the  IIR-M  configuration.  This  leaves  48  satellites  in  the  program  to  which  added  capabilities 
applied  at  the  time. 

21  See  Chapter  Four  for  a  more  comprehensive  discussion  of  the  cost-estimating  issues  asso¬ 
ciated  with  TSPR  and  other  acquisition  reform  measures. 
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annual  satellite  buys.  All  of  the  $700  million  in  growth  in  2002-2005 
is  attributable  to  the  quantity  increases  from  33  to  40  and  additional 
changes  in  acquisition  strategy. 

When  we  combine  the  two  lines  directly  associated  with  the  IIF 
satellite  program  on  Figure  2.5,  we  observe  a  net  increase  of  $438  mil¬ 
lion  for  a  program  that  will  procure  two  fewer  satellites  and  that  has 
yet  to  deliver  a  completed  satellite  ten  years  after  the  original  contract 
award. 

The  transfer  of  36  satellites  (the  program  quantity  was  reduced 
from  69  to  33  between  1999  and  2001)  and  future  ground  system 
capability  from  the  current  program  to  the  future  GPS  III  and  OCX 
segments  (purple  line)  resulted  in  $3,370  million  in  cost  underrun. 
This  entire  amount  represents  the  transfer  of  work  scope  to  a  newly 
created  follow-on  program  and  was  made  possible  in  large  part  by  the 
extraordinary  lengthening  in  the  average  life  expectancy  of  the  Block 
IIA  and  IIR  satellites. 

The  cumulative  procurement  variance  for  IIR  satellite  design  fixes 
and  technology  development  problems  (red  line)  is  just  over  $100  mil¬ 
lion.  All  of  this  growth  took  place  before  2003,  which  makes  sense 
because  all  IIR  satellites  were  delivered  by  the  end  of  2001. 

The  cumulative  procurement  variance  for  the  OCS  ground  seg¬ 
ment  (gray  line)  is  about  $200  million.  Growth  in  this  element  was 
small  until  2004,  at  least  partially  because  of  the  functionality  trans¬ 
ferred  out  of  the  program  to  the  future  OCX  segment  at  the  end  of 
2001.  We  believe  that  most  of  the  growth  in  this  effort  was  reported  in 
the  modernization  variances  through  1999  and  that  some  of  the  growth 
has  not  yet  been  reported  because  the  work  is  far  from  complete.  Infor¬ 
mation  provided  from  our  interviews  strongly  suggests  there  will  be 
substantial  additional  cost  growth  in  the  OCS  segment  in  the  future. 

About  $160  million  in  cumulative  procurement  cost  underrun 
was  not  identified  with  specific  program  segments  or  activities  (light 
blue  line).  These  variances  were  mostly  associated  with  inflation  adjust¬ 
ments,  reprogramming  of  funds  out  of  the  program,  and  program  wide 
support  function  estimate  changes. 

We  were  not  able  to  identify  specific  cost  growth  in  the  IIR-M 
effort.  The  effort  was  not  mentioned  in  a  single  one  of  the  102  procure- 
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ment  cost  variances.  It  is  likely  that  much  of  the  cost  growth  before 
2002  is  in  the  funding  removed  from  the  program  when  the  follow- 
on  GPS  III  and  OCX  efforts  were  created  and  associated  funding  was 
transferred  out  of  the  current  program  at  the  end  of  2001. 22  After  2001 
the  cost  growth  is  most  likely  buried  within  the  reduced  number  of 
satellites  to  be  modernized,  which  declined  from  12  to  eight  without 
associated  cost  estimate  adjustments.  Wherever  this  cost  growth  is,  we 
believe  that  it  will  continue  because  the  work  is  not  complete.23  In 
addition,  according  to  our  interviews,  the  contractor  has  requested  and 
been  granted  reimbursement  for  several  hundred  million  dollars  more 
in  overruns.  These  sources  of  cost  growth  are  not  reflected  in  the  data 
extending  to  the  end  of  2005. 

The  changes  in  the  development  plus  procurement  program  esti¬ 
mates  over  time  by  RAND  cost  growth  categories  are  shown  in  Figure 
2.6.  Cost  growth  and  underrun  in  each  subcategory  are  summed.  Nine 
subcategories  in  all  apply  to  the  program:  three  in  the  Errors  category, 
five  in  the  Decisions  category,  and  the  Financial  category.  Of  these 
nine,  four  caused  substantial  positive  cost  growth,  and  four  caused  sub¬ 
stantial  negative  cost  growth. 

The  program  experienced  about  $3,870  million  in  cost  growth, 
with  about  $1,440  million  in  the  Errors  subcategory  of  cost  estimates, 
$330  million  in  the  Errors  subcategory  of  technical  issues,  $240  mil¬ 
lion  in  the  Decisions  subcategory  of  schedules,  and  $1,860  million  in 
the  subcategory  of  requirements.  The  subcategory  of  quantity  domi¬ 
nated  the  cost  underrun,  accounting  for  $2,860  million  of  the  total 
$5,240  million  in  cost  underrun.  Decisions  associated  with  inter-  or 
intra-program  transfers,  mostly  development  funding  in  the  form 
of  transferring  capability  to  the  GPS  III  and  OCX  future  segments, 


22  This  is  done  by  underreporting  the  costs  associated  with  the  portion  of  the  program  trans- 
ferred  out  and  using  the  balance  of  that  value,  the  retained  funds,  to  “get  well”  on  cost 
growth  within  the  remaining  program  content. 

As  of  December  2005,  three  of  the  eight  IIR-M  satellites  had  been  delivered;  one  had  been 
launched. 
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Figure  2.6 

Cumulative  Variances  in  GPS  Development  and  Procurement  over  Time 
Using  RAND  Methodology 
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accounted  for  $880  million  in  cost  underrun,  and  inflation  changes 
accounted  for  the  $480  million  in  cost  underrun.  The  remaining  $1,020 
million  in  cost  underrun  came  from  the  subcategory  of  affordability, 
resulting  from  dozens  of  incidents  where  the  GPS  program  became 
a  “bill  payer”  for  other  programs.  Variances  in  the  GPS  SARs  were 
described  in  such  a  way  that  the  Errors  subcategory  of  schedule  was  not 
used,  though  undoubtedly  it  should  apply  to  some  subset  of  the  total 
variance  in  decisions. 

GPS  Cost  Variance  Bottom  Line 

The  development  and  production  contract  for  the  GPS  IIF  follow-on 
satellites  was  awarded  in  1996  under  aTSPR  arrangement  to  a  new  con- 
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tractor  at  an  “optimistic”  low  price.  During  the  mid-1990s,  awarding 
contracts  with  these  characteristics  was  common  and  a  product  of  the 
overall  defense  budget  environment  and  acquisition  reform  culture  of 
the  time.  This  acquisition  approach  added  much  higher  uncertainty  to 
these  satellites’  cost,  schedule,  and  technical  performance  outcomes. 

The  GPS  program  was  used  as  a  bill  payer  for  other  programs 
that  were  in  financial  need  throughout  the  ten-year  period.  In  addi¬ 
tion,  the  contract  EAC  for  the  OCS  modernization  increased  from 
$12  million  in  April  1996  to  $747  million  in  December  2005.  This 
illustrates  the  ever-increasing  requirements — especially  between  1998 
and  2001 — that  complicated  both  the  space  and  ground  segments  and 
drove  much  of  the  cost  growth.  A  strong  argument  can  be  made  that  a 
new  MS  B  should  have  been  declared  with  the  new  APB  of  late  2001 
and  early  2002. 

If  the  GPS  II  and  IIA  satellites  from  the  1980s  had  not  lasted  con¬ 
siderably  longer  than  expected  and  the  GPS  III  satellites  and  associated 
OCX  had  not  been  moved  into  a  future,  separate  program,  the  GPS 
program  would  not  have  been  the  bill  payer  that  it  was.  In  essence,  the 
program  unexpectedly  inherited  a  capability  that  allowed  it  to  delay 
launching  replacement  satellites.  At  the  same  time,  it  was  able  to  push 
off  onto  its  successor  program  many  of  the  capabilities  that  were  added 
as  new  requirements  during  the  ten-year  period  covered  in  our  analysis. 
Because  of  the  modernization  difficulties  in  all  the  program’s  current 
segments — the  IIR-M,  IIF,  and  OCS — it  would  have  experienced  mis¬ 
sion  deficiencies  if  these  added  capabilities  had  been  time-critical. 


Comparing  Cost  Variances  in  the  Two  Space  Programs 

The  relative  fractions  of  cost  growth  distributed  among  the  RAND 
variance  categories  for  the  two  programs  are  shown  in  Figure  2.7.  Some 
similarities  between  the  programs  include  the  following: 

•  Cost  estimate  errors  account  for  a  substantial  portion  of  cost 
growth. 

•  Quantity  was  cut  substantially  to  reduce  total  program  value. 
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Figure  2.7 

Comparison  of  Cost  Growth  Contributors,  by  the  RAND  Cost-Variance 
Categorization  Methodology 
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•  Decisions  to  add  requirements  and  change  schedules  were  sources 
of  cost  growth. 

•  Technical  difficulties  were  errors  with  significant  associated  cost 
growth. 

•  Inflation  was  a  source  of  cost  underrun. 


Figure  2.7  does  not  tell  the  whole  story.  Other  similarities  between 
the  two  programs  include  the  following: 

•  Both  programs  experienced  poor  outcomes  from  the  portion  of 
their  program  awarded  under  a  TSPR  contract. 

•  Both  experienced  huge  values  of  total  variance — cost  growth  and 
cost  underrun — over  the  ten-year  period,  $9.1  billion  in  GPS  and 
$9.8  billion  in  SBIRS. 

•  Both  were  “saved”  by  satellites  from  their  predecessor  pro¬ 
grams  that  have  remained  operational  substantially  longer  than 
expected. 
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While  both  programs  experienced  substantial  cost  growth  in  con¬ 
tent-adjusted  terms,  there  were  significant  differences,  including  those 
listed  below: 

•  Initial  and  final  program  values  were  quite  different:  $3.7  billion 
to  $10.2  billion  in  SBIRS-High;  $7.8  billion  to  $6.4  billion  in 
GPS. 

•  SBIRS-High  experienced  extremely  high  cost  growth  (by  histori¬ 
cal  standards)  attributable  to  the  Errors  category. 

•  GPS  experienced  more-typical  cost  growth  in  Errors  and  in  the 
Decisions  subcategory  of  requirements. 

•  SBIRS-High  SARS  were  slow  to  recognize  cost  growth  in  the 
program’s  first  four  years;  GPS  SARS  consistently  recognized  cost 
growth  during  that  same  time  frame. 

•  Except  for  quantity  reductions  in  both  programs,  SBIRS-High 
experienced  little  cost  underrun;  GPS  experienced  much  more. 

•  The  Air  Force  spent  much  more  than  planned  on  SBIRS-High 
over  the  ten-year  period  but  less  than  planned  on  GPS. 

•  SBIRS-High  experienced  very  little  requirements  changes  (by 
historical  standards);  GPS  experienced  dramatic  requirements 
changes. 


CHAPTER  THREE 


Technical  Risk  Assessment  Relating  to  Cost 
Estimation  for  SBIRS-High  and  GPS 


Introduction 

In  this  chapter,  we  analyze  the  technical  risk  assessments  in  the  cost¬ 
estimating  processes  for  the  SBIRS-High  and  GPS  modernization  pro¬ 
grams  (OCS,  IIR-M,  and  IIF)  to  gain  insights  into  factors  that  may 
have  contributed  to  the  underestimation  of  costs.  Several  studies  on 
national  security  space  acquisition  have  cited  technical  risks  as  one  of 
the  key  factors  that  contribute  to  cost  growth.1  Technical  risks,  such 
as  immature  technologies  or  a  compressed  testing  schedule,  lead  to 
technical  difficulties  that  could  eventually  result  in  failures  in  meeting 
the  technical  performance.  As  a  result,  redesigns  and  rework  may  be 
required,  which  slow  the  progress  of  the  program  and  cause  schedule 
slips  and  cost  growth.  It  is  not  surprising  that  underestimation  of  tech¬ 
nical  risks  was  one  of  the  key  factors  that  contributed  to  cost  estima¬ 
tion  errors  in  SBIRS-High  and  GPS,  as  the  SAR  analysis  in  Chapter 
Two  showed. 

Identifying  potential  technical  difficulties  associated  with  devel¬ 
oping  a  system  is  a  critical  component  in  the  cost-estimating  process.  It 
is  equally  critical  to  quantify  the  potential  cost  uncertainty  in  the  cost 


1  GAO,  Defense  Acquisitions:  Despite  Restructuring,  SBIRS  High  Program  Remains  at  Risk 
of  Cost  and  Schedule  Overruns,  GAO-04-48,  October  2003;  Defense  Science  Board,  The 
Future  of  the  Global  Positioning  System,  Washington,  D.C.:  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics,  October  2005;  and  Booz  Allen  Hamil¬ 
ton,  “Space  Systems  Development  Growth  Analysis,”  briefing,  Los  Angeles,  Calif.,  August 
2,2002. 
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estimates  due  to  technical  risks.  Understanding  the  technical  risks  of  a 
program  and  how  they  contribute  to  its  costs  is  of  paramount  impor¬ 
tance.  Arena,  Younossi,  et  al.  (2006)  discuss  risk  and  uncertainty  in 
detail  and  explore  how  risk  should  be  treated  in  cost  estimation.2  Here 
we  focus  on  detailed  technical  risk  assessment  processes  from  SBIRS- 
High  and  GPS.  The  objective  of  this  analysis  is  to  identify  broad  les¬ 
sons  learned  from  the  two  programs  to  extract  some  principles  that  can 
be  applied  to  technical  risk  assessments  in  cost-estimating  processes  in 
general.  These  general  principles  can  be  used  to  enhance  the  techni¬ 
cal  risk  assessment  processes  in  either  existing  programs  or  future  pro¬ 
grams  to  improve  the  cost-estimating  capability. 

We  begin  with  an  overview  of  the  technical  issues  that  the  SBIRS- 
High  and  GPS  programs  have  experienced  and  discuss  what  may  have 
caused  these  technical  difficulties.  These  technical  issues  provide  some 
insight  into  what  types  of  technical  risks  might  have  been  missed  or 
underestimated  throughout  the  development  phases.  We  then  examine 
how  technical  risks  were  assessed  and  then  translated  into  potential 
effect  on  costs  in  these  two  programs.  We  analyze  the  data,  the  meth¬ 
ods,  and  the  processes  (personnel,  information  flow,  etc.)  used  in  the 
assessments  to  identify  potential  factors  that  may  have  contributed  to 
underestimation  of  the  types  of  technical  risks  that  the  SBIRS-High 
and  the  GPS  programs  experienced  and  to  underestimation  of  cost 
implications.  We  conclude  with  a  summary  of  observations  and  recom¬ 
mendations  based  on  the  analysis. 


Overview  of  Technical  Issues  in  SBIRS-High  and  GPS 

SBIRS-High 

SBIRS-High  is  being  acquired  in  two  increments.  Increment  1  is  an 
upgrade  to  the  ground  segment  for  operating  the  DSP  satellites.  It  con¬ 
solidates  the  DSP  and  other  facilities  into  a  single  mission  control  sta- 


2  Mark  Arena,  Obaid  Younossi,  Lionel  Galway,  Bernard  Fox,  John  C.  Graser,  Jerry 
Sollinger,  Felicia  Wu,  and  Carolyn  Wong,  Impossible  Certainty:  Cost  Risk  Analysis  for  Air 
Force  Systems ,  Santa  Monica,  Calif.:  RAND  Corporation,  MG-415-AF,  2006. 
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tion.  Increment  2  includes  HEO  sensors,  to  be  put  on  separate  host 
satellites,  and  GEO  satellites,  as  well  as  an  upgrade  to  Increment  1  to 
operate  the  HEO  and  the  GEO  satellites. 

Since  the  inception  of  the  program,  SBIRS-High  has  encountered 
a  host  of  technical  problems  that  eventually  led  to  schedule  slips  and 
cost  growth.  Early  in  the  program,  SBIRS-High  experienced  technical 
difficulties  with  sensor  development;  examples  include  the  sensor  chip 
assembly  needed  for  sensor  detector  arrays  and  pointing  control  assem¬ 
bly  software  development.  The  control  gyro  reference  assembly  also 
failed  during  its  life  testing  and  had  to  be  fixed.  In  1999,  Increment  1 
failed  its  developmental  and  operational  testing  and  continued  to  expe¬ 
rience  problems  with  ground  software  that  resulted  in  a  two-year  slip. 
In  2001,  a  major  design  change  occurred  late  in  the  design  phase.  The 
GEO  design  had  to  add  a  12-foot  sunshade  for  off-axis  solar  radiation 
rejection.  The  prior  design  allowed  off-axis  solar  radiation  to  impinge 
on  the  focal  plane  array  when  the  sensor  line  of  sight  is  near  the  sun. 
This  problem  degraded  sensor  performance  and,  if  left  unaddressed, 
would  have  led  to  a  failure  to  meet  the  KPP.3 

Once  the  program  entered  the  testing  phase  for  the  HEO  sensor, 
multiple  technical  problems  were  found.  In  2000,  the  HEO  flattener 
lens  failed  during  the  first  random  vibration  test.  The  lens  came  out 
of  its  mounts  due  to  design  deficiencies.  Then  the  corrector  lens  failed 
during  the  second  test  in  2001.  These  failures  required  redesign  and 
rework,  which  added  to  cost  growth. 

After  the  program  restructure  in  2002,  serious  electromagnetic 
interference  (EMI)  problems  were  found  during  the  integration  of  the 
first  HEO  sensor  with  the  host  bus.  The  sensor  had  148  EMI  frequen¬ 
cies  that  did  not  meet  the  EMI  tolerance  requirements,  which  could 
have  adversely  affected  the  operations  of  the  host  satellite.4  Making 
design  modifications  to  correct  this  problem  added  one  year  to  the 


3  Timothy  Bonds,  Richard  Mesic,  Jon  Grossman,  Gary  McLeod,  Joel  Kvitky,  David 
Vaughan,  Rosalind  Lewis,  Manuel  Cohen,  Robert  Leonard,  James  Bonomo,  Lionel  Galway, 

Mel  Eisman,  and  John  Graser,  Assessing  Selected  Space-Based  Infrared  System-High  (SBIRS- 
High)  System  Risks,  Santa  Monica,  Calif.:  RAND  Corporation,  May  2003.  Government  pub¬ 
lication;  not  available  to  the  general  public. 
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schedule.  After  the  first  HEO  sensor  delivery,  SBIRS-High  continued 
to  experience  technical  issues,  for  example,  in  GEO  signal  process¬ 
ing  software  development  and  HEO-2  payload  software  qualification 
testing. 

GPS  IIR-M,  IIF  Modernization,  and  OCS 

The  modernization  of  HR  and  IIF  faced  some  difficulty  with  imple¬ 
menting  the  new  secure  military  (M-code)  signal.  The  contractor 
assumed  that  the  National  Security  Agency  (NSA)  already  had  a  chip 
that  had  the  M-code  algorithm.  However,  it  turned  out  that  the  NSA 
had  neither  the  chip  nor  the  algorithm.  The  contractor  had  to  work 
with  the  NSA  in  a  special  access  environment  to  develop  the  algorithm, 
which  added  a  number  of  complexities.5 

IIR-M  had  another  major  technical  issue  with  a  new  circuit  board 
that  was  required  for  the  M-code.  IIR-M  used  a  held  programmable 
gate  array  (FPGA)  commercial  chip,  and  the  FPGA  had  a  reliability 
problem  that  delayed  the  program  by  six  to  eight  months.  FPGA  was 
a  new-technology  item,  and  its  failure  mode  was  not  well  understood. 
Normally,  an  application-specific  integrated  circuit  (ASIC)  is  used,  but 
an  ASIC  takes  longer  to  program.  Because  of  a  schedule  problem  and 
the  general  emphasis  on  commercial  off-the-shelf  technologies  (COTS), 
the  IIR-M  program  decided  to  use  the  commercial  FPGA. 

Unlike  IIR-M,  the  IIF  modernization  program  used  an  ASIC 
chip  for  the  M-code,  but  it  experienced  other  technical  difficulties. 
The  contractor  tried  to  fit  everything  on  one  ASIC  and  spent  much 
time  optimizing  the  design.  Eventually,  IIF  ended  up  with  two  ASIC 
chips,  which  added  complexity  in  development  to  deal  with  the  inter¬ 
face  between  the  two  chips.  Additionally,  the  M-code  implementation 
on  IIR-M  and  IIF  turned  out  to  be  incompatible,  and  the  IIF  design 
had  to  be  changed.  The  government  believes  that  the  IIF  design  did  not 
function  properly,  although  the  contractor  disputed  this. 

There  have  also  been  technical  issues  during  integration,  assembly, 
and  testing  (IA&T)  in  IIF  and  IIR-M  due  to  late  deliveries  from  sup¬ 
pliers  and  quality  issues  with  some  parts.  In  some  cases,  specifications 


5  It  is  U.S.  government  policy  that  all  encryption  must  be  done  through  the  NSA. 
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had  not  properly  flowed  down  to  the  subcontractors,  and  less-stringent 
commercial  standards  were  used.  IIF  has  also  been  experiencing  prob¬ 
lems  with  developing  test  plans.  The  engineers  at  one  of  the  contractor’s 
sites  were  responsible  for  developing  the  test  plans  and  delivering  them 
to  another  site  for  IA&T.  However,  the  IA&T  staff  could  not  execute 
the  test  plans  because  the  plans  did  not  have  specific  detailed  proce¬ 
dures.  At  the  time  of  this  study,  the  contractor  had  proposed  that  the 
IA&T  facility  develop  the  test  procedures. 

Delivery  of  OCS  capabilities  has  continued  to  be  delayed  since 
the  beginning  of  the  program.  Currently,  OCS  operates  the  GPS  con¬ 
stellation  as  block  IIA  satellites.  Due  to  the  continuing  delays  in  capa¬ 
bility  delivery,  the  software  elements  that  are  necessary  for  operating 
the  flexible  power  and  the  new  signals  on  IIR-M  and  IIF  have  been 
removed  from  the  program.  OCX,  the  new  control  segment  program, 
will  be  implementing  these  capabilities.  As  a  result,  OCS  will  operate 
IIR-M  and  IIF  as  HR  satellites  when  OCS  is  complete. 

Early  in  the  program,  OCS  was  directed  to  maximize  the  use 
of  COTS  because  it  was  assumed  that  use  of  COTS  software  prod¬ 
ucts  would  save  significant  costs.  However,  OCS  experienced  difficul¬ 
ties  with  modifying  multiple  COTS  software  systems  and  integrating 
them.  Changing  much  of  the  COTS  software  was  challenging  because 
there  was  no  design  documentation;  small  changes  in  inputs  and  out¬ 
puts  translated  into  large  rewrites  of  the  software  because  engineers 
lacked  knowledge  about  the  details  of  the  COTS  software  that  was  to 
be  used  for  OCS.  Software  reliability  also  became  an  issue  because  the 
requirement  for  software  reliability  for  a  national  security  application 
is  much  higher  than  that  for  the  commercial  world.  In  addition,  there 
were  obsolescence  issues  with  the  COTS  during  development.  Keeping 
up  with  COTS  upgrades  added  to  the  technical  problems  because  any 
upgrade  to  any  software  product  required  a  rework  on  other  products 
due  to  the  large  number  of  interfaces. 

Potential  Causes  of  Technical  Difficulties  in  Development  of  SBIRS- 
High  and  GPS 

According  to  our  interviews  and  our  literature  review,  these  technical 
problems  had  multiple  causes.  In  SBIRS-High,  some  of  these  technical 
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problems  have  been  attributed  to  inadequate  systems  engineering,  an 
aggressive  schedule,  TSPR,  overly  aggressive  adoption  of  commercial 
standards  and  practices  for  military  applications,  a  lack  of  process  con¬ 
trols  at  both  prime  contractors  and  subcontractors,  and  the  reduction 
in  the  acquisition  workforce. 

Increment  1  ran  into  technical  difficulties  partly  due  to  overreli¬ 
ance  on  COTS  software  products  and  software  reuse.  The  program 
underestimated  the  technical  risks  associated  with  COTS.  The  prob¬ 
lem  with  the  off-axis  solar  radiation  on  the  GEO  design  was  discovered 
when  the  contractor  produced  a  better  thermal  model  after  the  critical 
design  review  (CDR).  It  is  not  clear  why  the  problem  was  discovered  so 
late  in  the  design  phase. 

On  the  EMI  matter,  our  interviews  suggest  that  it  was  a  process 
failure.  The  EMI  requirements  were  very  stringent  because  SBIRS- 
Eligh  is  a  multiple-payload  spacecraft.  There  was  an  EMI  design  review 
in  2000,  and  there  was  a  concurrence  on  the  EMI  control  plan.  How¬ 
ever,  the  subcontractor  did  not  implement  the  plan.  Another  factor 
that  may  have  contributed  to  the  EMI  problem  is  that  no  contractor 
was  clearly  responsible  for  integrating  the  HEO  sensor  with  the  host 
bus.  The  SBIRS  prime  contractor  was  responsible  for  the  HEO  sensor 
only. 

Some  technical  aspects  were  difficult  to  forecast  early  in  the  pro¬ 
gram.  For  a  system  as  complex  as  SBIRS-High,  system  integration  and 
flight  and  ground  software  turned  out  to  be  much  more  difficult  than 
estimated  in  part  because  of  a  lack  of  a  historical  analogue.  The  report 
from  the  SBIRS-High  Independent  Review  Team  in  2001  stated  that 
the  complexity  of  developing  engineering  solutions  to  meet  system 
requirements  was  not  well  understood  by  program  and  contracting 
officials. 

Some  of  the  technical  difficulties  experienced  in  IIR-M  and  IIF 
have  been  attributed  to  inadequate  contractor  capabilities  and  under¬ 
estimation  of  the  complexities  of  working  through  NSA.  For  IIF,  addi¬ 
tional  factors  such  as  lack  of  domain  knowledge  on  the  contractor’s 
part  and  instability  of  the  contractor’s  program  infrastructure,  may 
have  contributed  to  the  technical  problems.  The  contractor  moved 
development  and  production  work  from  one  location  to  another  on 
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several  occasions,  which  led  to  staffing  issues  and  high  turnover  rates. 
This  degraded  organic  capability  to  develop  the  system. 

The  Defense  Science  Board  (DSB)  report  on  GPS  in  2005  stated 
that  many  of  the  technical  problems  that  OCS  faced  since  the  begin¬ 
ning  of  the  program  have  been  attributed  to  substantial  use  of  COTS 
and  multiple  contractor  changeovers.6  Our  interviews  also  indicated 
that  the  contractors’  lack  of  domain  knowledge  and  poor  program 
management  may  have  contributed  to  some  of  the  technical  problems. 

In  general,  both  programs  significantly  underestimated  technical 
risks  and  their  potential  effect  on  costs.  At  SBIRS-High  source  selec¬ 
tion,  the  government’s  technical  assessment  was  not  thorough  enough 
to  capture  the  technical  risks  associated  with  system  integration  and 
software.  Technical  experts  focused  on  identifying  and  assessing  risks 
associated  with  key  technologies  and  underestimated  the  integration 
effort  required  for  a  complex  system  like  SBIRS-High.  The  experts  also 
did  not  focus  on  the  ground  software.  Flight  software  was  underesti¬ 
mated  because  of  lack  of  historical  perspective  (i.e.,  the  DSP  had  no 
flight  software).  Those  responsible  for  OCS  underestimated  technical 
risks  associated  with  using  COTS  for  controlling  a  unique  system  such 
as  GPS. 

Our  interviews  also  indicated  that  technical  risks  may  have  been 
underestimated,  and  these  risks  had  ripple  effects  due  to  the  complex 
interrelationships  of  various  components  and  subsystems.7  Both  the 
SBIRS-High  and  GPS  IIF  programs  experienced  technical  problems 
that  led  to  a  “standing  army”  effect;  that  is,  the  production  and  IA&T 
staff  were  essentially  placed  on  standby  because  of  delays  in  develop¬ 
ment  from  unanticipated  technical  issues.  For  instance,  the  HEO  EMI 
problem  caused  a  major  disruption  to  IA&T  while  the  EMI  problem 
was  being  corrected.  In  GPS  IIF,  the  original  program  was  already  in 
production  when  the  modernization  decision  was  made.  When  techni¬ 
cal  issues  in  development  of  the  modernized  IIF  delayed  the  develop¬ 
ment  schedule,  the  contractor  had  to  carry  production  people  at  a  lower 


6  Defense  Science  Board,  2005. 

7  By  ripple  or  cascade  effect,  we  mean  that  a  technical  problem  triggered  a  chain  of  other 
technical  problems. 
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work  rate  until  development  was  finished.  The  GPS  IIF  program  may 
have  also  underestimated  risks  in  technical  executability  due  to  inad¬ 
equate  contractor  capabilities,  an  inadequate  industrial  base,  and  the 
complexities  associated  with  working  with  the  NSA. 

A  review  of  the  technical  issues  in  the  SBIRS-High  and  GPS 
programs  indicates  that  technical  risks  associated  with  COTS  and 
software  reuse,  contractor  capabilities,  systems  integration  of  a  com¬ 
plex  system,  and  potential  ripple  effects  were  unanticipated  and  hence 
underestimated. 

Technical  Risk  Assessment  and  Quantification  Processes  on  the 
SBIRS-High  and  GPS  Programs 

The  SBIRS  and  GPS  programs  did  not  always  conduct  technical  risk 
assessments  to  support  cost  estimates.  In  this  monograph,  we  spe¬ 
cifically  address  the  processes  used  at  key  points  in  the  programs:  in 
SBIRS-High  at  source  selection  and  at  the  2002  and  2006  EACs,  and 
in  GPS  at  the  2001  program  office  estimate  (POE). 

At  source  selection,  the  SBIRS-High  program  office  performed 
cost  estimates  of  the  proposed  contractors’  designs  that  included  tech¬ 
nical  risk  assessments.  Since  source  selection,  the  SBIRS-High  pro¬ 
gram  office  has  completed  several  full-up  EACs,  which  included  tech¬ 
nical  assessments  in  2002  and  2006  to  support  program  re-baselines. 
Other  EACs  were  based  on  contractor’s  estimates  without  a  technical 
risk  assessment  on  the  program  office’s  part. 

At  source  selection,  the  SBIRS-High  program  office  implemented 
the  Relative  Risk  Weighting  (RRW)  technique,  a  component  of  the 
Cost  Risk  Identification  and  Management  System  (CRIMS)  process, 
to  assess  technical  risks  and  to  determine  the  cost-probability  distribu¬ 
tions.  CRIMS  was  developed  by  the  Space  and  Mission  System  Center 
Cost  Directorate  (SMC/FMC)  in  the  early  to  mid-1990s.  Since  source 
selection,  the  CRIMS  process  has  not  been  implemented  in  SBIRS- 
High.  The  program  office  developed  different  risk  assessment  and 
quantification  methods  in  2001,  and  the  program  office  has  continued 
to  use  the  same  approach.  The  approach  is  somewhat  similar  to  the 
CRIMS  RRW  method  but  in  a  simplified  form. 
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For  OCS  and  GPS  IIF,  we  were  unable  to  retrieve  detailed  infor¬ 
mation  about  the  cost-estimating  process  at  the  beginning  of  the  pro¬ 
grams  in  1994  and  1996,  respectively.  According  to  our  interviews,  the 
contractors  generated  the  estimates  for  these  programs  before  2001. 
When  the  modernization  decision  was  made  in  2001,  formal  POEs  for 
IIR-M,  IIF,  and  OCS  were  completed.  In  recent  years  (the  past  three 
years  or  so),  there  has  been  no  formal  technical  risk  assessment  process 
to  support  cost  estimation  in  GPS  IIF,  IIR-M,  and  OCS.  For  IIR-M 
and  OCS,  this  is  partly  due  to  the  fact  that  these  programs  are  in  late 
ph  ases  of  execution.  The  program  office  primarily  relies  on  “actuals”  to 
conduct  earned  value  (EV)  analysis  for  cost  estimates  rather  than  on  a 
formal  technical  risk  assessment.  The  technical  risk  assessment  process 
in  GPS  has  changed  since  the  2001  POE,  and  it  primarily  applies  to 
the  program’s  future  segments — GPS  III  and  OCX.  GPS  IIF  had  only 
partially  implemented  the  current  methods  and  processes  in  its  EAC 
in  2006,  and  thus  we  will  not  discuss  GPS’s  current  processes  in  this 
monograph. 

Our  information  about  the  technical  risk  assessment  processes 
used  in  cost  estimation  came  from  multiple  sources.  We  interviewed 
SBIRS-High  SPO  personnel  and  systems  engineering  and  technical 
assistance  (SETA)  contractors  who  provide  cost-estimation  support,  as 
well  as  Aerospace  Corporation  technical  support  personnel.8  We  also 
interviewed  current  and  prior  GPS  SPO  personnel  and  received  doc¬ 
umentation  on  the  2001  POE  cost-estimating  processes.  The  infor¬ 
mation  about  the  risk  assessment  methods  and  processes  for  the  2001 
POE  is  solely  based  on  the  2001  POE  report  for  IIF  and  OCS.  We 
assumed  that  the  technical  risk  assessment  and  quantification  processes 
for  the  IIR-M  cost  estimates  were  similar,  if  not  identical,  to  those  for 
the  IIF  and  OCS  in  2001  because  the  same  SETA  support  personnel 
developed  the  POE  for  IIR-M  as  well. 

The  technical  assessments  that  supported  the  2002  and  2006 
EACs  had  five  risk  categories  and  five  uncertainty  levels.  Table  3.1 
shows  these  categories  and  the  criteria  for  each  uncertainty  level.  The 


8  The  Aerospace  Corporation  is  a  federally  funded  research  and  development  center 
(FFRDC)  that  provides  technical  support  to  various  SMC  programs. 


Table  3.1 

SBIRS-High  Risk  Survey  in  2002  and  2006  EAC 


Uncertainty  Level 

Risk  Category 

1 

2 

3 

4 

5 

Required 

technical 

advance 

(science) 

Nothing  new 

Minor 

modifications  only 

Major 

modifications 

At  state  of  the  art 

Beyond  state  of 
the  art 

Technology 

status 

(engineering) 

In  use 

Prototype  exists 

Under  development 

In  design 

Concept  stage 

Complexity 

Simple 

Somewhat 

Moderately  complex 

Highly  complex 

High  complexity 
and  uncertainty 

Personnel/ 

equipment 

Expert  personnel 
with  equipment 
with  excess 
capability 

Trained  personnel 
with  equipment 
with  demonstrated 
capability  for  this 
application 

Mixture  of  trained 
and  new  personnel 
with  equipment  that 
appears  to  have  the 
capability 
to  do  the  job 

Insufficiently  trained 
personnel  with 
equipment  with 
known  inadequacies 

Untrained, 
inexperienced 
personnel  with  no 
useful  equipment 

Schedule 

quality 

Duplicate  of  past 
OR 

schedule  more 
than  25% 
expanded 

Similar  to  past 

OR 

schedule  expanded 
greater  than 
optimum 

Validated 

OR 

near  optimum 
(adequate  slack  time) 

Inadequate  analysis 
OR 

schedule  compressed 
more  than  optimum 

Unknown  or 
unsupported 
schedule 

OR 

schedule  more  than 
25%  compressed 
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SBIRS-High  program  office  developed  the  definition  of  the  risk  cat¬ 
egories  and  the  criteria  for  the  uncertainty  levels  based  on  the  Max¬ 
well  risk  criteria  matrix  (MRCM).9  There  is  no  document  containing 
a  detailed  definition  of  the  categories  or  criteria  for  these  uncertainty 
levels.  However,  for  the  2006  EAC,  the  cost  estimators  trained  the 
assessors  on  this  risk  assessment  method  to  clarify  the  ground  rules  and 
assumptions  of  the  method  as  well  as  the  definitions  and  the  criteria  of 
the  risk  categories  and  uncertainty  levels. 

The  GPS  programs  took  three  approaches  to  technical  risk  assess¬ 
ment:  (1)  hardware  work  breakdown  structure  (WBS)  elements  (IIF 
hardware  and  OCS  hardware)  whose  cost  estimates  were  based  on  cost¬ 
estimating  relationships  (CERs),  (2)  hardware  WBS  elements  whose 
cost  estimates  were  analogy-based,  and  (3)  software  (IIF  flight  software 
and  OCS  software)  WBS  elements.  For  CER-based  hardware  WBS 
elements,  there  were  ten  risk  categories:  required  technical  advance, 
technology  status,  complexity,  dependence  on  other  WBS  items,  corpo¬ 
rate  experience  level,  manufacturing  precision,  reliability,  producibility, 
schedule  estimate,  and  schedule  duration.  The  definitions  of  these  cat¬ 
egories  and  the  extreme  ends  of  the  uncertainly  levels  (i.e.,  the  lowest 
and  the  highest)  were  documented.  This  risk  assessment  framework 
did  not  apply  to  the  analogy-based  hardware  WBS  elements.  Instead, 
technical  experts  directly  provided  the  lower-bound,  most  likely,  and 
upper-bound  multipliers  to  be  applied  to  the  costs  of  IIR  analogous  ele¬ 
ments  as  a  means  to  account  for  risk  factors. 

Technical  risk  assessment  for  software  elements  followed  the 
method  that  was  incorporated  in  a  commercial  cost-estimating  model 
called  SEER.  The  model  had  33  risk  categories,  which  are  referred  to 
as  complexity  attributes  in  SEER. 


9  The  program  office  had  SETA  support  for  cost  estimation  and  it  developed  the  risk  assess¬ 
ment  methodology.  MRCM  is  a  risk  assessment  framework  developed  by  F.  D.  Maxwell  of 
the  Aerospace  Corporation  in  the  mid-1980s.  The  matrix  identifies  risk-drivers  and  a  range 
of  risk  levels.  See  David  R.  Graham,  “New  Initiatives  in  Program  and  Project  Management,” 
NASA  Presentation,  2005  NASA  Cost  Estimating  Symposium,  New  Orleans,  La.,  April 
12-14,  2005,  for  more  detailed  information. 
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Potential  Factors  That  Contributed  to  Underestimation  of 
Technical  Risk  Assessment 

Based  on  our  analysis  of  the  key  components  in  the  technical  risk 
assessment  processes  in  the  SBIRS-High  and  GPS  programs — data 
sources,  technical  risk  assessment  and  quantification  methodologies, 
staff  involved  in  the  processes,  information  flow  among  the  staff — we 
found  that  the  following  factors  may  have  contributed  to  inadequate 
technical  risk  assessments: 

•  data  availability  and  quality  issues 

•  credibility  issues  in  technical  risk  assessments 

•  limitations  in  risk  quantification  methodology. 

Data  Availability  and  Quality  Issues 

We  found  that  incomplete  and  unreliable  contractor  data,  lack  of  up- 
to-date  integrated  information,  and  limited  visibility  into  contractors’ 
capabilities  in  the  SBIRS-High  and  GPS  programs  may  have  limited 
the  programs’  ability  to  identify  all  technical  risks. 

For  instance,  SBIRS-High  still  lacked  a  fully  integrated  man¬ 
agement  system  as  of  the  2006  EAC.  Integrating  the  schedules  of  the 
prime  contractor  and  the  subcontractor  has  been  an  ongoing  challenge 
because  of  the  two  schedule  software  products  that  they  use.  For  GPS 
IIF,  an  integrated  master  schedule  (IMS)  is  not  on  the  contract  as  a 
Contract  Data  Requirements  List  item.  The  contractor’s  schedule  is 
presented  to  the  SPO,  but  it  is  difficult  to  understand  and  evaluate. 

GPS  IIF  and  OCS  have  also  experienced  problems  with  their 
data-reporting  systems  for  earned  value.  EV  data  provide  insights  into 
technical  executability  issues  that  the  contractor  may  be  experiencing. 
Our  interviews  indicated  that  the  contractor  had  an  inadequate  long¬ 
term  plan  and  made  optimistic  claims  regarding  what  work  content 
had  been  accomplished.  As  a  result,  the  contractor’s  cost  performance 
index  and  schedule  performance  index  were  never  realistic,  and  its  EV 
data  appear  to  have  been  of  questionable  quality.  OCS  presented  an 
additional  challenge  for  accessing  subcontractors’  data.  As  a  part  of  the 
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contract,  the  prime  contractor  is  responsible  for  providing  its  data  but 
not  that  of  the  subcontractors. 

We  have  also  found  that  an  up-to-date  comprehensive  view  of 
the  programs  was  not  always  available  for  a  thorough  technical  assess¬ 
ment.  Integrated  information  on  the  program  is  necessary  to  be  able  to 
identify  interrelationships  among  risks  (e.g.,  how  technical  risk  in  one 
WBS  element  might  affect  other  WBS  elements)  such  as  ripple  effects. 
As  mentioned  earlier,  the  IMS,  which  provides  an  integrated  view  of 
the  schedule,  has  not  been  available  in  complete  form. 

Additionally,  SBIRS-High  and  GPS  did  not  conduct  integrated 
baseline  reviews  (IBRs)  regularly  even  though  these  programs  under¬ 
went  significant  changes  throughout  development.  SBIRS-High ’s  first 
IBR  was  held  in  1999  at  the  time  of  the  Joint  Estimate  Team  (JET) 
re-baseline.  When  the  program  was  re-baselined  in  2002,  an  IBR  was 
held  after  the  EAC.  Our  interviews  also  indicated  that  the  2002  IBR 
did  not  have  adequate  representation  from  all  the  major  disciplines 
(technical,  schedule,  and  program  experts).  When  the  program  was 
replanned  in  2004,  the  SPO  did  not  conduct  an  IBR  because  the  SPO 
claimed  that  the  shoulder-to-shoulder  review  that  took  place  with  the 
contractors  at  that  time  was  sufficient.10  For  the  most-recent  EAC  in 
2006,  an  IBR  preceded  the  EAC.  Because  the  GPS  programs  relied 
on  the  contractor’s  cost  estimates  prior  to  2001,  it  appears  that  they 
did  not  conduct  IBRs  during  that  period.  Since  2001,  IIR-M  has  not 
conducted  an  IBR,  and  the  most  recent  IBR  on  OCS  was  in  2003.  IIF 
conducted  an  IBR  in  2003,  and  IIF  was  preparing  for  another  IBR  in 
2006  at  the  time  of  this  research. 

The  SBIRS-High  SPO’s  visibility  into  the  contractor’s  work  was 
also  limited  as  a  consequence  of  TSPR.* 11  GPS  IIR-M  and  IIF  had  a  sim- 


10  In  the  shoulder-to-shoulder  process,  personnel  from  the  contractor  and  the  government  sit 
together  and  review  the  data  and  reconcile  any  differences  as  necessary. 

11  TSPR  is  explained  in  detail  in  Chapter  Four.  The  key  aspects  of  TSPR  are  the  follow¬ 
ing:  (1)  The  contractor  formulates  and  proposes  its  own  technical  design  solution  to  meet 
high-level  DoD  performance  requirements,  (2)  The  contractor,  with  minimal  government 
oversight  and  direction,  is  responsible  for  implementing  the  proposed  solution  through  the 
development  process.  (3)  The  contractor  is  relieved  of  what  are  assumed  to  be  costly  and 
cumbersome  reporting  requirements  to  reduce  burdensome  overhead  costs. 
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ilar  challenge  because  they  were  also  executed  in  a  TSPR-like  approach 
due  to  acquisition  reform.  In  this  environment,  technical  staff  reviewed 
only  high-level  documents,  and  they  were  discouraged  from  any  fur¬ 
ther  detailed  review  of  the  contractor’s  work. 

Before  TSPR,  the  level  of  oversight  was  much  more  significant.  For 
example,  government  personnel  typically  attended  tests  and  conducted 
plant  visits  to  the  prime  contractor  as  well  as  at  the  subcontractors’  sites 
to  evaluate  the  facilities,  the  personnel,  and  process  controls.  Critical 
design  reviews  were  carried  out  with  data  packages,  design  analysis 
reports,  risk  analyses,  and  mitigation  plans  in  a  formal  structured  way. 
The  TSPR  environment  removed  this  level  of  rigor  for  monitoring  and 
assessing  contractor’s  capabilities.  As  a  result,  the  program  offices  pos¬ 
sessed  limited  information  for  conducting  thorough  and  accurate  tech¬ 
nical  risk  assessments.12 

Factors  Reducing  the  Credibility  of  Technical  Assessments 

Technical  risk  assessments  are  based  on  expert  judgment  and  are  fun¬ 
damentally  subjective.  As  a  result,  the  credibility  of  the  assessments 
depends  on  the  experts’  level  of  expertise  and  the  level  of  subjectivity. 
We  concluded  that  the  technical  risks  in  the  SBIRS-High  and  GPS 
programs  may  have  been  underestimated  because  of 

•  inadequate  level  of  expertise 

•  lack  of  independence 

•  lack  of  bases  for  judgment 

•  flaws  in  elicitation  methods. 

Inadequate  Level  of  Expertise.  Because  technical  risk  assessment 
is  based  on  expert  judgment,  it  is  critical  that  technical  experts  with 
appropriate  domain  knowledge  assess  both  the  technical  content  and 
the  technical  executability  of  the  program.  Inadequate  levels  of  tech¬ 
nical  expertise  may  have  affected  the  quality  and  the  thoroughness 
of  the  technical  assessments  on  the  SBIRS-High  and  GPS  programs. 


12  See  Chapter  Four  for  a  more  comprehensive  discussion  of  the  historical  context  and  cost 
estimating  issues  associated  with  TSPR,  COTS  insertion,  and  other  acquisition  reform 


measures. 
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The  number  of  Aerospace  Corporation  personnel,  who  primarily  pro¬ 
vided  critical  technical  support  for  SBIRS-High,  had  been  significantly 
reduced  by  the  time  the  contract  was  awarded  in  1996.  By  the  late 
1990s,  the  Aerospace  support  staff  had  been  reduced  to  less  than  half 
of  its  original  size  in  1994.  This  was  due  to  the  assumption  that  TSPR 
would  require  much  less  in-house  technical  expertise  in  the  program 
office.  The  GPS  program  also  experienced  reductions  in  Aerospace  sup¬ 
port  due  to  acquisition  reform  in  the  1990s. 

In  the  SBIRS-High  2002  and  2006  EACs,  project  officers,  inte¬ 
grated  product  team  (IPT)  leaders  (e.g.,  space  IPT,  ground  segment 
IPT),  and — in  some  instances — Aerospace  technical  experts  conducted 
the  program  office’s  risk  assessments.  The  project  officers  were  primar¬ 
ily  junior  officers  and  in  general  had  limited  experience  and  expertise. 

EV  data  provide  insights  into  technical  risks  and  schedule  risks. 
But  the  EV  analysts  in  the  SBIRS-High  and  GPS  programs  lacked 
the  expertise  to  understand  the  content  of  the  work  and  what  the  data 
meant  in  terms  of  how  the  program  might  be  affected.  Because  of  the 
SPO’s  limited  resources,  they  received  limited  support  from  the  techni¬ 
cal  experts;  the  SPO  technical  workforce  was  focused  on  the  technical 
aspect  of  the  program  and  support  for  the  EV  analysts  was  minimal. 

Given  that  technical  risk  assessments  are  ultimately  used  to  quan¬ 
tify  cost  uncertainties,  a  thorough  and  accurate  technical  risk  assess¬ 
ment  requires  an  integrated  and  interactive  effort  between  the  techni¬ 
cal  and  cost  experts.  However,  there  was  insufficient  communication 
between  the  technical  and  cost  experts  in  the  risk  assessment  processes 
of  the  SBIRS-High  and  GPS  programs.  In  these  risk  assessment  pro¬ 
cesses,  the  technical  experts  filled  out  surveys  or  questionnaires,  and 
the  cost  estimators  translated  the  information  into  probability  distri¬ 
butions  without  any  feedback  to  the  technical  experts  on  their  assess¬ 
ments.  In  some  cases,  technical  experts  were  required  to  quantify  cost 
effects,  as  in  the  case  of  the  GPS  2001  POE  for  some  of  the  WBS  ele¬ 
ments.  Technical  assessments  with  inadequate  communication  between 
technical  and  cost  experts  appear  to  have  degraded  the  accuracy  of  the 
assessments  since  most  technical  experts  lack  cost  expertise. 

Lack  of  Independence:  Reliance  on  Contractor's  Assessments 
and  SPOs  as  Advocates.  Technical  risk  assessments  in  the  SBIRS-High 
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and  GPS  programs  lacked  independence  because  they  were  based  pri¬ 
marily  on  the  contractor’s  and  the  SPO  personnel’s  judgment,  with 
little  outside  disinterested  input.  Given  that  these  assessments  are  sub¬ 
jective,  there  is  a  possibility  of  biases  as  a  result  of  the  program  environ¬ 
ment  and  pressures  from  senior  leadership  in  the  SPO,  which  may  have 
caused  the  risks  to  be  inaccurately  assessed. 

During  the  SBIRS-High  source  selection,  budget  pressures,  the 
rushed  environment,  and  optimism  about  TSPR  and  “best  commercial 
practices”  may  have  played  a  role  in  influencing  the  technical  risk  assess¬ 
ment.  Little  uncertainty  about  the  technical  parameters  that  drive  the 
costs  was  assumed  in  assessing  configuration  risks.  For  example,  the 
cost  estimate  included  a  spacecraft  weight  growth  factor  of  17  percent 
compared  with  a  typical  historical  growth  factor  of  approximately  50 
percent.  Additionally,  there  was  no  software  growth  assumption  in  the 
cost  estimate.  It  was  assumed  that  the  contractor  design  included  a  suf¬ 
ficient  margin.  By  2002,  the  GEO  SV  dry  weight  had  grown  over  50 
percent,  and  the  software  equivalent  Software  Lines  of  Code  (SLOC) 
had  grown  by  about  250  percent. 

Our  interviews  indicate  that  the  environment  during  the  2002 
re-baselining  also  posed  similar  challenges  for  the  SBIRS-High  pro¬ 
gram.  Due  to  the  2002  Nunn-McCurdy  breach,13  there  was  significant 
institutional  pressure  to  keep  the  revised  cost  estimate  down  and  to  fix 
the  program  to  protect  it  from  the  real  possibility  of  cancellation.  This 
environment  may  have  influenced  the  judgment  of  the  contractor  and 
the  SPO  about  the  level  of  remaining  technical  risks  in  the  program. 
For  instance,  even  after  2002,  both  the  HEO  and  GEO  payload  devel¬ 
opment  efforts  encountered  a  series  of  additional  technical  issues  that 
led  to  further  schedule  delays  and  cost  growth.  In  the  risk  survey  that 
supported  the  2002  EAC,  all  the  GEO  and  HEO  payload  WBS  ele¬ 
ments  were  rated  at  the  moderate  risk  level  or  lower.  Only  seven  WBS 
elements  out  of  about  80  received  a  risk  rating  of  high  (triangular  prob- 


13  A  “Nunn-McCurdy”  unit  cost  breach  occurs  when  an  MDAP  experiences  an  increase  of 
at  least  15  percent  in  program  acquisition  unit  cost  or  average  procurement  unit  cost.  A  unit 
cost  breach  of  more  than  15  percent  requires  formal  notification  to  Congress.  A  unit  cost 
breach  of  25  percent  or  more  requires  a  recertification  of  the  program  by  the  Secretary  of 
Defense,  which  includes  the  development  and  certification  of  new  unit  cost  estimates. 
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ability  distribution’s  high-end  multiplier  of  2.0),  and  they  were  mostly 
ground  segment  elements.  It  is  unclear  if  an  independent  assessment 
would  have  yielded  a  higher  risk  assessment.14 

Part  of  the  SBIRS-High  risk  assessment  processes  in  2002  and 
2006  relied  on  the  contractor’s  assessment.  The  SPO  used  the  contrac¬ 
tor’s  point  estimates  and  technical  risk  assessments  as  the  baseline.  The 
contractor  presented  its  list  of  specific  risk  items,  and  the  SPO  worked 
with  the  contractor  to  adjust  the  assessment  either  by  adding  or  delet¬ 
ing  risk  items  and  by  adjusting  the  probability  of  occurrence  and  costs 
of  risk  items.  Because  the  baseline  risk  items  and  their  costs  were  those 
of  the  contractor,  the  risk  assessment  may  have  been  either  incomplete 
(i.e.,  all  risks  may  not  be  identified  thoroughly)  or  evaluated  in  a  more 
optimistic  light  (i.e.,  the  probability  of  occurrence  and  cost  were  under¬ 
estimated),  given  that  the  contractor  has  its  own  incentives  that  may 
bias  the  technical  assessment. 

Lack  of  Bases  for  Judgment  and  Rigor  in  Technical  Assessment. 

One  of  the  main  flaws  in  the  technical  risk  assessments  in  the  SBIRS- 
High  and  GPS  programs  was  that  acquisition  reform  and  use  of  COTS 
were  assumed  to  bring  cost  savings.  There  was  no  fact-finding  or  rigor¬ 
ous  analysis  to  support  the  technical  risk  assessment  of  these  practices 
and  products. 

Technical  risks  associated  with  TSPR  were  not  adequately  assessed 
at  SBIRS-High  source  selection.  The  TSPR  environment  and  the  focus 
on  COTS  led  to  removal  of  military  standards  and  specifications  and 
reduction  in  government  oversight.  This,  in  turn,  led  to  process  control 
and  quality  control  issues  that  resulted  in  serious  technical  problems, 
such  as  the  HEO  EMI  problem.  The  GPS  programs  experienced  simi¬ 
lar  problems  with  the  contractors.  There  was  no  fact-finding  or  histori¬ 
cal  perspective  to  support  the  assumed  benefits  of  acquisition  reform. 

Technical  risks  associated  with  COTS  were  assumed  to  be  low 
without  an  in-depth  assessment  of  the  implementation  of  COTS.  For 


14  An  independent  RAND  study  at  the  time  explicitly  noted  that  additional  technical  risks 
existed  beyond  those  identified  during  the  re-baseline  and  would  likely  result  in  additional 
technical  problems  and  cost  growth.  The  study  findings  were  published  in  Bonds  et  ah,  2003, 
but  this  report  was  not  released  to  the  general  public,  nor  was  it  widely  circulated  within  the 
Air  Force. 
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example,  implementing  COTS  software  for  GPS’s  telemetry,  tracking 
and  command  (TT&C)  turned  out  to  be  a  difficult  problem  because 
TT&C  for  GPS  is  unique  and  much  more  complex  than  for  com¬ 
mercial  satellites  (which  are  primarily  communications  satellites);  there 
are  more  satellites  to  control  in  the  GPS  constellation  and  GPS  has 
its  own  unique  infrastructure  to  control  its  satellites.  Similarly,  SBIR’s 
GEO  bus  was  assumed  to  be  low  risk  because  it  was  based  on  a  com¬ 
mercial  bus  (the  bus  was  assumed  to  be  a  recurring  cost  rather  than 
a  nonrecurring  cost).  However,  the  GEO  bus  experienced  significant 
configuration  changes  and  weight  growth  because  of  unique  military 
requirements.  Again,  no  rigorous  technical  risk  assessment  supported 
the  initially  assumed  low  risk. 

Flaws  in  the  Elicitation  Process.  Cost  analysts  often  draw  on  the 
expertise  of  engineers,  managers,  and  other  experts  for  subjective  prob¬ 
ability  distributions  to  represent  the  uncertainty  in  technical  and  other 
program  inputs  that  form  the  basis  of  the  cost  estimate.15  They  do  so 
to  account  for  the  uncertainty  that  results  from  the  lack  of  thorough 
understanding  of  the  technology,  the  impact  of  future  economic  con¬ 
ditions,  overoptimism  on  the  part  of  program  advocates,  and  the  like. 
Lack  of  clarity  in  the  elicitation  process  for  cost-probability  distribu¬ 
tions  (e.g.,  merely  surveying  expert  judgment  for  information),  com¬ 
bined  with  a  minimal  number  of  experts,  may  have  caused  variances  in 
experts’  judgment  and  created  opportunities  for  bias,  leading  to  unin¬ 
tended  misjudgments  about  technical  risks.  In  fact,  experts  often  dis¬ 
agree  and  are  prone  to  bias  just  as  laypersons  are.16  Lavallo  and  Kahne- 
man  argue  that  executives  routinely  exaggerate  the  benefits  of  major 
projects  and  discount  their  costs.  They  often  assume  “the  best  scenario” 
and  forget  to  account  for  the  difficulties  and  schedule  delays  that  are  an 


15  Lionel  A.  Galway,  Subjective  Probability  Distribution  Elicitation  in  Cost  Risk  Analysis:  A 
Review ,  Santa  Monica,  Calif.:  RAND  Corporation,  TR-410-AF,  2007. 

^  See  M.  Granger  Morgan,  “Choosing  and  Managing  Technology-Induced  Risk,”  IEEE 
Spectrum  (18)12,  1981,  and  William  D  Ruckelshaus,  “Risk,  Science,  and  Democracy,”  Issues 
in  Science  and  Technology  1(3),  1985. 
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inherent  part  of  complex  projects.17  Arena,  Younossi,  et  al.  (2006)  pro¬ 
vide  a  detailed  discussion  of  the  key  biases  relevant  to  cost  estimation. 

Recall  that  in  SBIRS-High  and  GPS,  technical  experts  assessed 
technical  risks  by  rating  the  WBS  elements’  risk  levels  in  various  risk 
categories.  The  risk  categories  and  the  criteria  for  the  various  uncer¬ 
tainty  levels  guided  the  assessment.  However,  in  SBIRS-High ’s  2002 
and  2006  risk  assessments,  the  definitions  of  the  risk  categories  and  the 
criteria  for  the  uncertainty  levels  lacked  clarity  and  precision  (see  Table 
3.1).  No  formal  documentation  with  clear  definitions  and  instructions 
guided  the  technical  experts.  As  a  result,  each  evaluator  may  have  had 
his  or  her  own  interpretations  of  what  types  of  uncertainties  should  be 
considered  in  each  risk  category,  based  on  their  individual  experience. 
The  evaluators’  definitions  of  risk  categories  and  criteria  for  the  uncer¬ 
tainty  levels  may  have  been  inconsistent  with  what  the  cost  estimators 
intended.  Because  technical  experts  focus  on  technical  performance, 
they  may  overlook  certain  types  of  technical  uncertainties  that  can 
have  large  cost  implications.  In  the  SBIRS-High ’s  2006  EAC,  however, 
training  on  the  risk  assessment  method  for  the  risk  evaluators  became 
mandatory  and  provided  better  guidelines  for  the  evaluators. 

The  2001  GPS  risk  assessment  process  had  similar  shortcomings. 
For  example,  only  the  extreme  endpoints  of  the  uncertainty  scale  (i.e., 
0  and  10)  were  defined  in  the  instruction.  One  evaluator’s  criteria  for 
a  score  of  5  may  have  differed  from  another’s  criteria,  which  may  have 
also  differed  from  the  cost  estimator’s  criteria.  GPS’s  2001  elicitation 
process  had  another  weakness  in  that  only  one  expert  provided  the 
assessment  per  WBS  element.  Given  the  lack  of  clarity  and  precision 
in  the  risk  survey,  this  further  created  an  opportunity  for  inaccurate 
technical  risk  assessment  for  cost-estimating  purposes. 

Limitations  in  Risk  Quantification  Methodology 

Even  if  the  technical  risk  assessments  are  thorough  and  accurate,  accu¬ 
rately  translating  the  subjective  assessments  into  cost  implications  in  a 
quantitative  way  is  yet  another  challenging  step  in  the  cost  estimation 


17  Dan  Lovallo  and  Daniel  Kahneman,  “Delusions  of  Success:  How  Optimism  Undermines 
Executives’  Decisions,”  Harvard  Business  Review ,  July  2003. 
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process.18  We  observed  that  the  following  factors  in  the  risk  quantifica¬ 
tion  processes  in  SBIRS-High  and  GPS  may  have  led  to  underestima¬ 
tion  of  cost  estimates: 

•  potential  errors  in  selecting  applicable  cost-probability 
distributions 

•  limitations  of  the  risk  quantification  methods  for  capturing  large 
potential  cost  growth. 

Errors  in  Selecting  Applicable  Cost-Probability  Distributions. 

At  SBIRS-High  source  selection  and  during  formulation  of  the  GPS 
2001  POE,  the  normalized  cost-probability  distributions  were  unique 
to  each  WBS  element  to  account  for  each  WBS  element’s  unique  char¬ 
acteristics  that  drive  the  relationship  between  the  risk  level  and  cost 
uncertainty.  However,  the  quantification  method  used  for  SBIRS-High 
since  2001  does  not  lend  itself  to  developing  tailored  distributions.  It 
uses  four  normalized  triangular  distributions  that  correspond  to  four 
risk  levels.  That  is,  WBS  elements  in  the  same  risk  level  category  have 
the  same  normalized  cost-probability  distribution.  Using  a  standard 
set  of  probability  distributions  to  quantify  risks  for  all  WBS  elements 
may  under-  or  overestimate  risks  in  some  WBS  elements.  For  exam¬ 
ple,  a  “high-risk”  hardware  element  may  experience  a  different  cost- 
uncertainty  profile  than  a  “high-risk”  software  element.  The  accuracy 
of  these  probability  distributions  is  also  uncertain  because  the  distri¬ 
butions  were  selected  based  on  a  SAR  analysis  of  all  DoD  programs 
conducted  in  1992. 19  It  is  uncertain  if  the  average  data  from  all  DoD 
programs  applies  to  space  programs,  especially  given  the  increases  in 
the  level  of  complexity  in  space  programs  since  the  time  of  the  SAR 
analysis. 


18  Arena,  Younossi,  et  al.,  2006,  present  a  comprehensive  assessment  of  various  risk  analysis 
methods  and  a  policy  prescription  for  how  to  address  risk  in  cost  estimation. 

19  The  Analytic  Sciences  Corporation  (TASC)  conducted  the  study.  It  identified  three  dis¬ 
tributions  (high  points  at  1.1,  1.4,  and  2)  but  the  SBIRS-High  SPO  added  the  fourth  distri¬ 
bution  to  account  for  the  very  high  risk  software  WBS  element.  See  Shishu  Gupta,  David 
Olsen,  David  Hudak,  and  Jennifer  Keenan,  “Cost  Risk  Analysis  of  the  Strategic  Defense 
System,”  TR-9042-2,  Revision  1,  Arlington,  Va.:  TASC,  1992,  for  more  details. 
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The  overall  risk  level  and  ultimately  the  endpoints  of  the  triangu¬ 
lar  cost-probability  distributions  depend  on  the  numerical  weightings 
of  the  risk  categories  and  the  numerical  scale  values  of  the  uncertainty 
levels  (see  Table  3.1).  There  are  some  weaknesses  in  how  these  param¬ 
eters  were  developed  in  both  the  SBIRS-High  and  GPS  programs.  In 
the  2002  and  2006  SBIRS-High  processes,  the  relative  weighting  on 
the  risk  categories  was  dispensed  with  (i.e.,  all  risk  categories  had  equal 
weighting)  and  the  uncertainty  level  scale  was  assumed  to  be  linear 
from  1  to  5  for  all  risk  categories.  The  technical  experts  who  evaluated 
the  risk  level  did  not  provide  any  input  regarding  the  relative  impor¬ 
tance  of  the  risk  categories  or  the  intensity  scale  of  the  uncertainty 
levels.  The  cost  estimators  calculated  the  overall  risk  level  as  the  aver¬ 
age  of  all  the  uncertainty  levels  in  all  risk  categories.  As  a  result,  the 
overall  technical  assessment  could  have  been  skewed  if  one  of  the  risk 
drivers  (e.g.,  technology  maturity  level)  had  greater  cost  implications 
than  others  (e.g.,  personnel,  equipment).  The  equal  weighting  was  also 
applied  even  if  a  risk  category  might  have  not  been  applicable  to  certain 
WBS  elements.  In  the  2006  EAC,  the  risk  evaluators  were  instructed 
to  rate  the  uncertainty  level  as  ’’lowest”  if  a  risk  category  did  not  apply 
to  the  WBS  element  being  evaluated.  As  a  result,  the  contribution  from 
the  nonapplicable  risk  category  would  cause  the  overall  risk  score  to  be 
underestimated.  The  GPS  risk  assessment  method,  which  was  similar 
to  the  SBIRS-High ’s  2002  and  2006  processes,  also  neglected  the  rela¬ 
tive  importance  of  the  risk  categories,  hence  potentially  introducing 
errors  at  the  quantification  step. 

Additionally,  for  the  GPS  2001  POE  and  SBIRS-High ’s  2002 
and  2006  processes,  cost  estimators  determined  the  uncertainty  level 
scales  without  any  input  from  technical  experts  and  without  any  basis. 
For  example,  there  was  no  basis  for  selecting  a  linear  scale  rather  than 
an  exponential,  logarithmic,  or  some  other  mathematical  function.  In 
the  CRIMS  method,  technical  experts  determined  the  weighting  and 
the  scales,  but  these  parameters  were  still  based  on  expert  judgment 
(and  thus  subjected  to  the  weaknesses  discussed  in  previous  sections), 
and  they  lacked  any  sound  analytical  basis  for  establishing  the  mathe¬ 
matical  relationship  between  those  parameters  and  the  cost-probability 
distributions. 
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Throughout  both  programs,  none  of  the  risk  quantification  results 
were  revisited  or  cross-checked  using  other  methods  for  validation  or  as 
a  “sanity  check.”  The  lack  of  rigorous  analysis  to  support  the  selection 
of  cost-probability  distributions  and  the  lack  of  validation  may  there¬ 
fore  have  contributed  to  underestimating  the  level  of  cost  uncertainty 
in  these  programs. 

Methodology  Limitations  in  Quantifying  Potentially  Large  Cost 
Growth.  One  of  the  limitations  in  using  the  statistical  method  in  the 
cost  estimate  is  that  technical  risks  are  only  quantified  as  the  variance 
of  the  cost-probability  distribution  (the  spread  of  the  distribution).  As 
a  result,  if  the  point  estimate  is  significantly  underestimated,  the  cost 
estimate  may  still  be  underestimated  even  at  a  reasonably  high  con¬ 
fidence  level.  This  approach  may  not  adequately  capture  the  effect  of 
technical  uncertainties  that  can  lead  to  large  cost  growth. 

For  the  SBIRS-High’s  2002  and  2006  EACs,  some  of  the  techni¬ 
cal  risks  were  deterministically  quantified  to  adjust  the  point  estimates. 
The  point  estimate  included  specific  risk  items  whose  probabilities  of 
occurrence  were  estimated  to  be  75  percent  or  greater.  However,  this 
approach  has  some  weaknesses:  There  is  no  solid  analytical  rationale 
for  using  75  percent  as  the  threshold,  and  the  rating  of  the  probability 
of  occurrence  is  subjective.  For  the  2002  EAC,  the  total  cost  of  the 
risk  items  whose  probabilities  of  occurrence  were  below  75  percent  was 
fairly  significant.  Their  full  value  was  2.5  times  that  of  the  risk  items 
above  the  threshold.  Given  the  high  level  of  subjectivity  involved  in 
this  approach,  a  more-rigorous  analysis,  cross-check,  or  sensitivity  anal¬ 
ysis  on  the  assessment  of  the  probability  levels  and  the  threshold  level 
might  have  resulted  in  a  different  outcome. 

Modeling  risk  correlation  is  another  challenge  in  quantifying  risk. 
As  mentioned  earlier,  the  SBIRS-High  cost  estimate  at  source  selection 
did  not  include  risk  correlation,  and  thus  the  cost  uncertainty  due  to 
technical  risks  may  have  been  underestimated.  In  later  processes,  risk 
correlation  was  added  in  the  cost  models.  For  the  SBIRS-High  EACs 
in  2002  and  2006,  relatively  large  correlations  were  assumed  among 
all  WBS  elements.  However,  cost  estimators  determined  these  values, 
and  there  were  no  analyses  or  standards  to  support  the  quantification 
of  correlations.  For  GPS,  correlation  was  captured  by  quantifying  the 
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commonality  between  the  WBS  elements  based  on  expert  judgment. 
Correlation  was  also  introduced  as  one  of  the  risk  categories  (“depen¬ 
dence  on  other  WBS  elements”)  to  assess  the  effect  on  cost  uncertain¬ 
ties.  Again,  the  values  used  to  quantify  the  correlations  were  not  sub¬ 
stantiated  by  any  analysis  or  historical  database.  Furthermore,  the  risk 
quantification  methods  used  by  these  programs  model  risk  correlation 
such  that  it  affects  the  variance  of  the  total  cost-probability  distribu¬ 
tion.  As  a  result,  risk  correlation  that  leads  to  large  cost  growth,  such 
as  cascading  effects  and  “standing  army  effects,”  may  not  have  been 
adequately  captured,  even  with  conservative  values  for  risk-correlation 
coefficients. 

Capturing  risk  correlations  at  a  detailed  level  is  complex.  Although 
selecting  an  arbitrary  number  for  correlation  is  better  than  setting  the 
correlation  to  zero,  some  validation  of  the  correlation  values  is  needed 
to  estimate  the  cost.  Additionally,  a  separate  analysis  may  be  required  to 
explicitly  quantify  technical  risk  correlations  that  could  lead  to  a  large 
cost  growth,  requiring  adjustments  to  the  point  estimate  as  necessary. 


Summary  Observations 

Many  challenges  are  associated  with  identifying  technical  risks,  eval¬ 
uating  them,  and  translating  them  into  cost  effects.  It  is  inherently 
difficult  to  forecast  future  technical  problems  and  any  resulting  cost 
growth  they  might  incur.  Program  managers  often  assume  “the  best 
scenario”  and  forget  to  account  for  the  difficulties  and  schedule  delays 
that  are  an  inherent  part  of  complex  projects.  A  large  number  of  fac¬ 
tors  drive  technical  risks,  and  many  of  them  are  interrelated.  However, 
accuracy  and  confidence  in  technical  risk  assessments  can  be  improved 
with  rigorous  processes  and  methods. 

Technical  risk  assessments  in  the  SBIRS-High  and  GPS  programs 
lacked  rigor,  partly  because  of  limited  resources  and  acquisition  reform 
that  reduced  in-house  expertise,  as  well  as  optimism  and  overreliance 
on  contractors’  capabilities.  The  cost  estimates  did  not  always  include  a 
formal  technical  risk  assessment  and  quantification  to  support  the  cost 
estimates.  The  technical  risk  assessments  that  were  conducted  lacked 
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rigorous  fact-finding  analysis  to  support  the  assessments  of  technical 
content  (e.g.,  COTS)  and  technical  executability  (e.g.,  TSPR). 

The  level  of  up-to-date  data  on  technical  baseline,  integrated 
schedule,  and  earned  value  was  inadequate  for  thorough  up-to-date 
identification  and  evaluation  of  technical  risks.  Key  data  sources  were 
unreliable  or  lacked  regular  updates.  The  SBIRS-High  and  GPS  SPOs’ 
visibility  into  contractor’s  capabilities  was  also  limited. 

Because  technical  risk  assessments  are  fundamentally  subjective, 
certain  aspects  of  the  technical  assessment  methodologies  and  processes 
may  have  created  opportunities  for  unintended  misjudgment  about  the 
level  of  technical  risk.  The  SPOs’  in-house  level  of  expertise  (both  the 
number  of  experts  and  their  level  of  experience)  was  inadequate.  The 
risk  assessment  process  may  have  been  influenced  by  a  lack  of  insti¬ 
tutional  independence:  It  was  conducted  mainly  by  SPO  personnel 
or  contracted  support  personnel  (including  FFRDCs),  and  the  prime 
system  contractor,  none  of  whom  were  disinterested  observers.  No 
truly  independent  technical  assessment  supplemented  the  SPOs’  tech¬ 
nical  risk  assessment  in  the  early  phases  of  the  programs.  Institution¬ 
ally  independent  cost  estimates,  such  as  those  conducted  by  the  OSD 
Cost  Analysis  Improvement  Group  (CAIG)  on  SBIRS-High  after  the 
first  Nunn-McCurdy  breaches,  depended  largely  on  the  same  techni¬ 
cal  risk  assessments  as  the  SPO  and  contractor  estimates.  Not  surpris¬ 
ingly,  those  cost  estimates  proved  to  be  more  realistic,  but  they  also 
did  not  adequately  reflect  the  true  amount  of  technical  risk  that  still 
remained  in  the  program.  Limited  communications  between  the  tech¬ 
nical  experts  who  conducted  the  assessment  and  the  cost  estimators 
who  used  the  information  may  have  introduced  further  unintended 
errors  in  the  risk-assessment  process  as  well  as  in  the  process  of  translat¬ 
ing  the  technical  risks  in  terms  of  cost  implications. 

There  were  limitations  in  the  methodologies  for  quantifying  the 
technical  risks  in  terms  of  their  cost  implications,  which  may  have  con¬ 
tributed  to  cost-estimating  errors.  The  justification  for  selecting  appli¬ 
cable  cost-probability  distributions  and  correlation  models  lacked  solid 
technical  grounds.  The  quantification  methods  had  limitations  in  cap- 
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turing  factors  that  could  contribute  to  potentially  large  cost  growth. 
Given  that  any  method  has  some  limitations,  some  form  of  validation 
or  cross-check  with  a  different  method  or  model  could  have  minimized 
potential  cost-estimating  errors. 


CHAPTER  FOUR 


Other  Nontechnical  SBIRS-High  and  GPS 
Contributing  Factors 


Overall  Case  Study  Findings  on  the  Causes  of 
Cost-Estimation  Errors 

This  chapter  examines  some  of  the  major  programmatic  factors  that 
contributed  to  “inappropriate  cost  estimates”  or  cost-estimation  errors 
identified  by  the  RAND  SAR  categorization  methodology  in  the 
SBIRS-High  and  GPS  programs.  It  presents  our  overall  findings  from 
our  two  case  studies  and  provides  some  more-detailed  discussion  and 
documentation  of  those  findings,  with  the  key  exception  of  those  areas 
related  to  the  technical  risk-assessment  and  quantification  process, 
which  are  discussed  in  the  preceding  chapter.  It  also  describes  the  con¬ 
text  in  which  acquisition  decisions  were  being  made. 

As  noted  previously,  the  true  causes  of  cost-estimation  errors  are 
difficult  to  discern  with  confidence  solely  from  SARs.  Based  on  our 
interviews,  we  concluded  that  the  RAND  SAR  cost-variance  categori¬ 
zation  methodology  most  likely  underestimates  the  scale  of  cost  growth 
that  should  be  attributed  to  “technical  issues”  in  the  Errors  category,  at 
least  for  SBIRS-High  and  GPS.  This  is  mainly  because  the  SARs  are 
often  ambiguous  about  the  sources  of  cost  growth.  The  RAND  cost- 
variance  methodology  generally  assigns  cost  variances  to  the  “technical 
issues”  subcategory  only  when  they  are  clearly  called  out  in  the  SAR 
as  causing  the  cost  variance.  It  became  clear  from  our  extensive  inter¬ 
views  with  program  experts  that  technical  issues  were  responsible  for 
far  more  cost  variance  (which  we  placed  in  the  cost  estimation  errors 
subcategory)  than  the  SARs  identify  directly. 
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Analysis  of  our  extensive  interviews  with  numerous  senior  gov¬ 
ernment  and  contractor  support  officials  who  were  involved  in  the 
cost-estimating  and  technical  risk-assessment  processes  on  these  two 
programs  between  1996  and  2005  led  us  to  the  overarching  conclu¬ 
sion  that  the  entire  acquisition  process  as  whole  resulted  in  optimistic  cost 
estimates.  Multiple,  complex,  and  interrelated  causes  led  to  optimistic 
cost  estimates,  including  organizational,  budgetary,  and  institutional 
factors.  We  isolated  the  following  three  main  categories  of  causal  fac¬ 
tors  that  led  to  overoptimistic  cost  estimates  and  other  cost  errors  for 
both  programs: 

•  TSPR  and  acquisition  reform  as  implemented  did  not  achieve  the 
expected  benefits  in  both  programs. 

•  The  cost-estimating  process  was  organizationally  too  closely  asso¬ 
ciated  with  bureaucratic  interests  that  held  advocacy  positions, 
making  independent,  disinterested  cost  analysis  more  difficult. 

•  Inadequate  cost-estimation  and  risk-assessment  methods  and 
models  were  used  in  both  programs. 

In  the  remainder  of  this  chapter,  we  examine  the  first  two  points 
in  greater  detail.  Chapter  Three  documented  the  third  finding. 


Acquisition  Reform,  TSPR,  and  the  Abdication  of  Rigorous 
Program  Oversight 

Both  SBIRS-High  and  the  original  GPS  IIF  development  programs 
were  formally  launched  in  1996  at  the  height  of  the  Clinton  adminis¬ 
tration’s  attempt  to  implement  a  wide  range  of  acquisition  reform  mea¬ 
sures,  many  of  which  were  intended  to  bring  about  radical  change  in 
the  way  DoD  had  historically  developed  and  acquired  major  weapon 
systems  and  military  space  systems.  Nearly  all  senior  officials  we  inter¬ 
viewed  identified  acquisition  reform  in  general,  and  the  TSPR  reform 
concept,  in  particular,  as  root  causes  contributing  to  the  initial  lack  of 
credible  cost  estimates  on  both  the  SBIRS-High  and  GPS  moderniza- 
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tion  programs.1  The  fundamental  problems  for  cost  analysts  with  the 
way  acquisition  reform  measures  were  implemented  in  our  case  study 
programs  in  the  1990s  were  fourfold.  TSPR  and  acquisition  reform  had 
the  following  effects: 

•  They  eliminated  the  requirements  for  contractors  to  provide  the 
necessary  cost  and  technical  data  to  support  independent  govern¬ 
ment  assessments. 

•  They  rationalized  the  reduction  in  manpower  and  functional 
expertise  in  the  SPOs,  which  hindered  the  SPOs’  ability  to  moni¬ 
tor  and  assess  contractor  performance. 

•  They  fostered  an  environment  that  facilitated  SPOs’  abdication 
of  their  traditional  role  of  independently  assessing  contractor 
performance. 

•  They  led  to  overestimation  of  cost  savings  from  the  insertion 
of  COTS  hardware  and  software  and  other  acquisition  reform 
measures. 

The  remainder  of  this  section  discusses  the  background  of  acqui¬ 
sition  reform  and  TSPR  in  the  1990s  and  how  they  affected  SBIRS- 
High  and  GPS. 

Following  the  collapse  of  the  Soviet  Union  in  1989  and  the  end  of 
the  Cold  War,  the  decline  in  the  defense  budget  accelerated  and  policy¬ 
makers  looked  forward  to  reaping  the  benefits  of  a  large  “peace  divi¬ 
dend”  that  was  expected  to  result  from  the  emergence  of  a  more-stable, 
lower-threat  world  environment.  Defense  planners  became  increasingly 
interested  in  reforming  the  defense  acquisition  system  to  make  it  more 
efficient  and  less  costly  and  to  increase  positive  symbiotic  interactions 
with  the  civilian  economy  and  commercial  technology  development. 

Reform  efforts  dramatically  accelerated  and  were  broadened  to 
the  whole  federal  government  after  the  election  of  President  Bill  Clin¬ 
ton  in  1992.  In  March  1993,  President  Clinton  established  an  inter¬ 
agency  task  force,  later  called  the  National  Performance  Review  (NPR), 
headed  up  by  Vice  President  A1  Gore  and  aimed  at  making  “the  entire 


i 


We  explain  the  TSPR  concept  in  greater  detail  later  in  this  section. 
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federal  government  less  expensive  and  more  efficient.”2  The  NPR  pro¬ 
duced  a  massive  overview  report  with  384  recommendations  presented 
to  the  president  in  September  1993. 3  It  was  accompanied  by  no  fewer 
than  15  “government  systems”  reports,  including  one  on  “Reinventing 
Federal  Procurement,”  and  20  major  studies  on  reforming  the  most 
important  federal  agencies,  including  the  DoD.  The  DoD  report  noted 
that  “the  DOD  acquisition  system  must  undergo  a  fundamental,  top- 
to-bottom  transformation.” 

The  report  put  forward  three  guiding  principles  for  acquisition 
reform.  The  first  focused  on  radically  reducing  the  regulatory  and  gov¬ 
ernment  oversight  burden  placed  on  defense  contractors  by  DoD  and 
encouraging  the  adoption  of  commercial  best  business  practices.  The 
second  and  third  principles  aimed  at  strengthening  the  overall  national 
and  defense  industrial  bases  through  much  greater  emphasis  on  the 
use  of  commercial  technologies  and  products  by  DoD  and  defense 
contractors.4 

William  Perry,  the  new  Secretary  of  Defense  in  the  Clinton 
administration,  became  a  strong  advocate  of  far-reaching  reform  for 
the  defense  acquisition  system.  In  early  1994,  he  launched  a  wide  range 
of  major  defense  acquisition  reform  initiatives  aimed  at  increasing  effi¬ 
ciency,  reducing  costs,  and  enhancing  civil-military  industrial  and 
technological  integration.5  In  accordance  with  the  principles  enunci¬ 
ated  in  Vice  President  Gore’s  NPR  reports,  Secretary  Perry’s  reform 
efforts  emphasized  reduction  in  the  government  regulatory  and  over¬ 
sight  activities  imposed  on  defense  contractors,  adoption  of  best  busi¬ 
ness  commercial  practices  by  both  the  government  and  defense  con- 


2  “Remarks  by  President  Clinton  Announcing  the  Initiative  to  Streamline  Government, 
March  3,  1993,”  cited  in  A  Brief  History  of  Vice  President  Al  Gore’s  National  Partnership  for 
Reinventing  Government  During  the  Administration  of  President  Bill  Clinton,  1993—2001. 

3  Office  of  the  Vice  President,  National  Performance  Review,  From  Red  Tape  to  Results:  Cre¬ 
ating  a  Government  that  Works  Better  and  Costs  Less,  Washington,  D.C.,  September  1993b. 

4  Office  of  the  Vice  President  Department  of  Defense,  Accompanying  Report  of  the  National 
Performance  Review,  Part  1  of  2,  Washington,  D.C.,  September  1993a. 

5  See  William  Perry,  Secretary  of  Defense,  Acquisition  Reform:  A  Mandate  for  Change,  State¬ 
ment  to  U.S.  Congress  House  Armed  Services  Committee  and  Government  Affairs  Com¬ 
mittee,  February  9,  1994. 
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tractors,  and  the  maximum  use  of  COTS  technologies  and  products 
in  defense  acquisition  programs.6  Darken  Druyun,  who  was  appointed 
Principal  Deputy  Assistant  Secretary  of  the  Air  Force  for  Acquisition 
and  Management  in  1993,  became  a  strong  advocate  of  acquisition 
reform,  wielding  unprecedented  influence  over  Air  Force  acquisition 
programs. 

Throughout  the  1990s,  OSD  and  the  services  promulgated  a 
bewildering  array  of  acquisition  reform  directives.  Probably  the  most 
important  thrust  of  the  overall  effort,  however,  was  the  goal  of  radically 
reducing  the  DoD  regulatory  and  oversight  burden  placed  on  industry. 
DoD  suppliers  have  historically  been  subjected  to  a  vast  and  complex 
body  of  unique  regulatory  and  oversight  requirements.  Acquisition 
reformers  argued  that  these  measures  were  largely  unnecessary  and  led 
to  significant  economic  inefficiencies  that  forced  DoD  to  pay  a  large 
“regulatory  premium”  for  goods  and  services.  Furthermore,  according 
to  the  reformers,  DoD  reporting  and  oversight  regulations  artificially 
separated  the  defense  industrial  base  from  the  rest  of  the  national  high- 
technology  industrial  base,  prevented  defense  contractors  from  increas¬ 
ing  efficiency  by  adopting  “best  commercial  practices,”  discouraged 
participation  of  “best-of-breed”  commercial  firms  in  defense  programs 
as  DoD  suppliers,  and  shut  off  DoD  from  easy  access  to  dramatic  new 
high-technology  developments  taking  place  in  the  commercial  sector. 
The  reformers’  solution  to  these  alleged  problems  was  to  reduce  or 
eliminate  special  DoD  regulations  and  oversight  as  much  as  possible. 

By  1996,  a  variety  of  major  statutory  and  regulatory  changes 
aimed  at  reducing  regulation  and  oversight  of  contractors  and  trans¬ 
ferring  developmental  responsibility  and  power  to  contractors  had 
coalesced  into  the  concept  called  Total  System  Performance  Respon- 


6  RAND  reports  that  investigate  and  evaluate  various  aspects  of  these  reform  efforts 
include  Mark  Lorell,  Julia  Lowell,  Michael  Kennedy,  and  Hugh  Levaux,  Cheaper,  Faster, 
Better ?  Commercial  Approaches  to  Weapons  Acquisition,  Santa  Monica,  Calif.:  RAND  Cor¬ 
poration,  MR-1147-AF,  2000;  Mark  Lorell  and  John  C.  Graser,  An  Overview  of  Acquisition 
Reform  Cost  Savings  Estimates,  Santa  Monica,  Calif.:  RAND  Corporation,  MR-1329-AF, 
2001;  and  Mark  A.  Lorell,  John  C.  Graser,  and  Cynthia  R.  Cook,  Price-Based  Acquisition: 
Issues  and  Challenges  for  Defense  Department  Procurement  of  Weapon  Systems,  Santa  Monica, 
Calif.:  RAND  Corporation,  MG-337-AF,  2005. 


76  Improving  the  Cost  Estimation  of  Space  Systems 


sibility.7  The  most  important  non-statutory  regulatory  changes  flowed 
from  Secretary  Perry’s  1994  memorandum  cited  above  that  banned  use 
of  military  specifications  and  standards  on  DoD  contracts  without  an 
explicit  waiver,  in  order  to  promote  the  use  of  COTS  technology  and 
parts  and  commercial  best-business  practices  by  defense  contractors. 
This  was  the  basis  for  the  move  away  from  detailed  technical  specifi¬ 
cations  in  DoD  Requests  for  Proposals  (RFPs),  toward  performance 
specifications  in  which  the  government  called  out  desired  high-level 
system  performance  outcomes  and  contractors  provided  the  technical 
design  solutions  to  meet  the  performance  outcomes.  In  summary,  the 
TSPR  acquisition  concept  brought  together  all  these  initiatives  into  a 
single  comprehensive  strategy.  The  key  aspects  of  the  TSPR  concept 
were  as  follows: 

•  The  contractor  formulates  and  proposes  its  own  technical  design 
solution  to  meet  high-level  DoD  performance  requirements. 

•  The  contractor,  with  minimal  government  oversight  and  direction, 
is  responsible  for  implementing  the  proposed  solution  through 
the  development  process. 

•  The  contractor  is  relieved  of  costly  and  cumbersome  reporting 
requirements  to  reduce  what  are  assumed  to  be  burdensome  over¬ 
head  costs.8 

TSPR  was  implemented  side  by  side  with  another  major  initiative 
called  Cost  As  an  Independent  Variable  (CAIV).  The  CAIV  concept 
placed  a  tremendous  emphasis  on  reducing  cost  and  providing  maxi¬ 
mum  value  to  the  government  for  the  taxpayer’s  dollar.  The  CAIV  con- 


7  The  most  important  statutory  changes  flowed  from  the  Federal  Acquisition  Streamlin¬ 
ing  Act  of  1994,  the  Defense  Acquisition  Management  Reform  Act  of  1995,  and  the  Federal 
Acquisition  Reform  Act  of  1996.  See  Lorell  and  Graser,  2001;  and  Michael  FI.  Anderson, 
A  Study  of  the  Federal  Government’s  Experiences  with  Commercial  Procurement  Practices  in 
Major  Defense  Acquisitions ,  Master  of  Science  in  the  Management  of  Technology  Thesis, 
Alfred  P.  Sloan  School  of  Management  and  The  School  of  Engineering,  Massachusetts  Insti¬ 
tute  of  Technology,  June  1997. 

8  The  catch  phrase  at  the  time  was  “insight,  not  oversight.”  For  a  more  extensive  discussion  of 
the  TSPR  concept,  see  Major  Henry  P.  Pandes,  “Total  System  Performance  Responsibility — 
Defense  Logistics,”  Air  Force  Journal  of  Logistics,  Summer  2001 . 
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cept  aimed  at  placing  as  much  emphasis  on  cost  as  had  been  tradition¬ 
ally  placed  on  system  performance.  The  goal  was  to  find  the  “knee  of 
the  curve,”  where  cost-effectiveness,  or  performance  for  the  dollar,  was 
maximized.  The  practical  effects  of  the  heavy  emphasis  on  CAIV,  how¬ 
ever,  translated  into  heavy  pressures  on  contractors  to  meet  demanding 
performance  requirements  at  much  lower — often  unrealistically  low — 
cost  compared  with  what  would  have  typically  been  thought  possible 
in  the  past.  The  acquisition  reformers  thought  contractors  could  do  this 
successfully  by  incorporating  much  more  advanced  commercial  tech¬ 
nology  and  COTS  components,  by  adopting  “best  business  practices” 
from  the  commercial  business  sector,  and  by  eliminating  the  allegedly 
costly  DoD  “regulatory  burden”  placed  on  defense  contractors.  Past 
RAND  studies,  as  well  as  this  study,  suggest  that  these  expectations  for 
cost  savings  were  not  well  thought  out  or  verified  by  senior  DoD  advo¬ 
cates  before  their  widespread  adoption  throughout  the  DoD  acquisi¬ 
tion  process.9 

A  key  aspect  of  the  TSPR  concept  was  a  dramatic  reduction  in 
contractor  data  and  documentation  reporting  requirements.  “Insight, 
not  oversight”  was  interpreted  as  eliminating  the  need  for  much  of 
the  large  quantity  of  traditional  formal  documentation  in  government 
format,  such  as  highly  detailed  Statements  of  Work  and  the  extensive 
data  reporting  requirements  traditionally  laid  out  in  formal  Contractor 
Data  Requirements  Lists.  As  a  result,  the  TSPR  concept  dramatically 
reduced  the  documentation  and  reporting  requirements  for  contrac¬ 
tors,  including  the  reporting  and  documentation  of  technical  and  cost 
data.  This  of  course  was  bound  to  have  a  significant  effect  on  the  SPO’s 
ability  to  undertake  traditional  cost  and  technical  risk  analysis.10 

Both  the  SBIRS-High  and  GPS  modernization  development  pro¬ 
grams  were  launched  in  a  TSPR  and  CAIV  environment.  The  SBIRS- 
High  development  effort  was  touted  with  considerable  fanfare  as  one  of 


9  See  Lorell  and  Graser,  2001. 

10  For  an  optimistic  discussion  of  the  early  phases  of  SBIRS-High  as  an  acquisition  reform 
streamlining  pilot  program,  see  Major  Jay  A.  Moody,  Achieving  Affordable  Operational 
Requirements  on  the  Space  based  Infrared  System  (SBIRS)  Program:  A  Model  for  Warfighter  and 
Acquisition  Success ?  Student  Research  Paper,  AU/ACSC/97-0548,  Air  Command  and  Staff 
College,  March  1997. 
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DoD’s  first  major  acquisition  reform  pilot  programs,  chosen  to  demon¬ 
strate  and  test  the  acquisition  reform  concepts  announced  by  Secretary 
Perry  in  his  1994  memorandum,  especially  TSPR,  CAIV,  and  COTS 
insertion.  For  example,  in  1995  SBIRS-High  was  one  of  the  first  DoD 
programs  to  employ  a  Single  Acquisition  Management  Plan  (SAMP), 
which  was  part  of  the  effort  at  a  dramatic  reduction  in  paperwork  and 
contractor  reporting  requirements.  In  the  case  of  SBIRS-High,  the 
SAMP  consolidated  20  documents  that  in  the  past  typically  would 
have  added  up  to  around  1,000  pages  into  a  SAMP  that  was  only  37 
pages  long.  Formerly  long  and  detailed  planning  documents  such  as 
the  Acquisition  Plan,  Acquisition  Strategy  Plan,  Test  and  Evaluation 
Master  Plan,  Acquisition  Program  Baseline,  Integrated  Logistics  Sup¬ 
port  Plan,  and  Risk  Assessment  were  radically  shortened  and  all  folded 
into  this  one  short  SAMP.  The  Cost  and  Financial  Management  section 
of  the  SBIRS-High  SAMP  took  up  a  mere  four  pages;  Program  Con¬ 
tent,  only  two  pages;  and  the  Program  Reviews  and  Oversight  section, 
only  three  pages.  In  contrast,  the  section  on  Acronyms  and  Related 
Programs  took  up  a  full  nine  pages  of  the  SAMP.  This  is  not  surprising, 
because  SBIRS-High  was  established  in  complete  conformity  with  the 
TSPR  concept,  where  government  oversight  and  guidance  were  to  be 
kept  to  a  minimum,  and  maximum  program  and  technical  responsibil¬ 
ity  were  to  be  transferred  to  the  contractor.11 

Major  reductions  in  the  DoD  workforce,  particularly  in  the  acqui¬ 
sition  field,  reinforced  the  Clinton  administration’s  push  for  acquisition 
reform,  founded  on  reduced  government  oversight  and  the  transfer  of 
greater  design  and  technical  responsibility  to  contractors.  In  1995,  in 
accordance  with  the  Clinton  administration’s  goal  of  streamlining  gov¬ 
ernment,  the  DoD  established  the  goal  of  reducing  its  workforce  by 
25  percent  by  the  end  of  FY00.  After  two  years,  the  DoD  acquisi¬ 
tion  workforce  had  already  been  reduced  by  almost  16  percent,  nearly 
two-thirds  of  the  required  five-year  goal.  As  one  Government  Account- 


11  See  James  Clausen  and  Leonard  Sadaukas,  Office  of  the  Deputy  Assistant  Secretary  of 
Defense,  Deputy  Chief  Information  Officer,  Blueprint  for  Establishing  Risk-based  Gover¬ 
nance  of  IT  Investments  in  a  Net-centric  Department  of  Defense,  Version  1.0,  13  April  2005, 
pp.  48-49. 
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ability  Office  (GAO)12  study  noted  in  1998,  DoD  was  reducing  its 
acquisition  workforce  “at  a  significantly  higher  rate  than  it  reduced  its 
overall  workforce.”13  Other  data  show  that,  from  1994  through  1997, 
the  SBIRS-High  SPO  workforce  declined  by  about  40  percent.  During 
the  same  period,  the  support  staffing  provided  by  FFRDCs  and  SETA 
contractors  at  the  SBIRS-High  SPO  went  down  by  over  50  percent.14 
This  is  important,  because  SMC  SPOs  have  historically  depended 
heavily  on  SETA  contractors,  such  as  Tecolote  Research,  and  FFRDCs, 
such  as  the  Aerospace  Corporation,  to  support  cost  and  technical  risk 
analyses. 

This  all  occurred  when  the  SBIRS-High  and  GPS  IIF  and  IIR-M 
programs  were  being  defined,  launched,  and  moved  through  initial 
full-scale  development.  Thus,  the  dramatic  reductions  in  SPO  person¬ 
nel,  which  took  place  beginning  in  1995,  coincided  and  reinforced 
the  trend  toward  reducing  oversight  and  transferring  technical  and 
cost  responsibility  to  the  contractor,  which  was  also  being  encouraged 
simultaneously  by  TSPR  and  other  acquisition  reform  measures. 

Adding  to  the  challenges  for  cost  analysts  was  DoD’s  sharply 
increased  emphasis  on  using  COTS  technology,  parts,  subsystems,  and 
software  on  satellites  to  reduce  costs  and  shorten  schedules.  Greater  use 
of  COTS  was,  of  course,  a  central  theme  of  Secretary  Perry’s  acquisi¬ 
tion  reform  initiatives.  Unfortunately,  experience  now  suggests  that  the 
use  and  integration  of  COTS  hardware  and  parts,  especially  software, 
has  proven  far  more  technically  challenging  than  originally  anticipated 
by  the  acquisition  reform  theorists.  As  in  the  case  of  other  acquisition 
reform  measures,  there  is  no  evidence  that  DoD  seriously  set  out  to  test 
and  validate  the  widely  held  assumptions  that  the  use  and  insertion 
of  COTS  components,  software,  and  technologies  would  dramatically 
reduce  costs.15 


12  Formerly  the  General  Accounting  Office. 

13  Government  Accountability  Office,  Defense  Acquisition  Organizations:  Status  ofWorkforce 
Reductions ,  Letter  Report,  GAO/NSIAD-98-161,  June  29,  1998. 

14  Booz  Allen  Hamilton,  2002,  p.  174. 

15  For  a  detailed  discussion  of  some  of  these  challenges,  see  Lorell  et  ah,  2000. 
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As  a  result,  we  found  that  the  two  most  pernicious  specific  aspects 
of  acquisition  reform  affecting  the  cost-estimating  process  for  our  case 
studies  were 

•  overoptimistic  initial  program  estimates  based  on  unsubstanti¬ 
ated  assumptions  regarding  savings  from  less  regulatory  oversight, 
the  use  of  COTS,  and  “best  business  practices” 

•  abdication  of  the  government  responsibility  and  capability  to  pro¬ 
vide  oversight  and  to  track  contractor  progress  and  keep  cost  esti¬ 
mates  current. 

Numerous  senior  officials  we  interviewed  told  us  that  huge  unsub¬ 
stantiated  savings  were  assumed  and  factored  into  the  initial  program 
estimates  for  SBIRS-High  and,  to  a  lesser  agree,  the  GPS  moderniza¬ 
tion  program.  One  interviewee  told  us  that  something  on  the  order 
of  $3  billion  in  savings  was  credited  to  TSPR  in  the  original  SBIRS- 
High  program  estimate.  Additional  savings  were  credited  to  the  use 
of  COTS.  Some  have  claimed  that  initial  cost-modeling  efforts  using 
standard  cost  assessment  tools  produced  much  higher  cost  estimates, 
but  these  were  rejected  by  senior  Air  Force  leadership  as  outdated  and 
unable  to  account  for  the  expected  acquisition  reform  savings.  Yet  none 
of  these  assumed  savings  from  acquisition  reform  and  TSPR  had  ever 
been  analytically  demonstrated  or  validated.  We  note  that  the  simulta¬ 
neous  consolidation  of  the  aerospace  industry  and  the  reduction  of  the 
government’s  acquisition  workforce  coalesced  with  the  above  factors  to 
create  an  environment  that  made  accurate  cost  estimation  particularly 
challenging. 

Even  more  unfortunate,  acquisition  reform  and  TSPR,  combined 
with  SPO  downsizing,  made  it  far  more  difficult  to  assess  the  realism 
of  contractor  cost  estimates  (as  well  as  technical  designs  and  risks)  and 
track  and  assess  contractor  performance  once  a  program  was  under 
way,  which  may  explain  why  SBIRS-High ’s  serious  problems  took  so 
long  to  be  recognized.  For  example,  during  the  source  selection  pro¬ 
cess  on  SBIRS-High,  acquisition  reform  permitted  the  competing  con- 
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tractors  to  submit  contractor-unique  work  breakdown  structures.16  It 
proved  difficult  to  correlate  these  with  the  standard  government  WBS 
guidelines.  Also  as  a  result  of  TSPR,  contractor  cost  estimates  were 
provided  down  only  to  the  relatively  high  level  of  WBS  Level  3,  which 
is  the  space  vehicle  level,  thus  providing  government  analysts  with  little 
insight  into  the  technical  and  cost  assumptions  behind  the  contractor 
cost  estimates. 

The  SBIRS-High  SPO  also  did  not  have  enough  personnel  or 
people  with  adequate  training  to  interpret  contractor  EV  reports  prop¬ 
erly  once  development  was  under  way.  The  contractors  reported  in 
their  own  company  formats,  designating  and  sometimes  shifting  their 
baselines,  thus  making  the  reports  difficult  for  government  analysts 
to  interpret.  TSPR  also  resulted  in  the  failure  to  clarify  a  common 
understanding  and  definition  of  categories  for  the  Contractor  Cost 
Data  Reports  at  source  selection.  Final  agreements  and  understandings 
in  this  area  did  not  occur  until  years  after  the  SBIRS-High  program 
launch. 

In  the  case  of  GPS  IIF  and  OCS,  acquisition  reform  and  “best 
business  practices,”  combined  with  SPO  downsizing,  appear  to  have 
contributed  to  the  failure  of  the  SPO  to  make  adequate  assessments 
of  the  contractor’s  capabilities  to  implement  the  program  and  to  track 
the  program’s  progress.  Major  developmental  and  production  issues 
emerged  in  these  programs  as  a  result  of  poor  execution  by  the  contrac¬ 
tor.  However,  the  SPO  had  little  warning  until  the  problems  became 
major,  because  it  was  neither  able  nor  willing  to  track  contractor  per¬ 
formance  closely.  In  addition,  the  heavy  acquisition  reform  empha¬ 
sis  on  COTS  insertion  also  appears  to  have  directly  contributed  to 
underestimation  of  the  cost  of  the  GPS  modernization  effort.  (Chap¬ 
ter  Three  discussed  the  technical  challenges  encountered  in  integrating 
COTS  hardware  and  software  in  much  greater  detail.)  Apparently  all 
these  challenges  added  significantly  to  cost  and  failed  to  produce  the 
expected  overall  savings. 


16  The  WBS  organizes  a  project  into  smaller  components  so  it  can  be  developed  and  man¬ 
aged  more  effectively. 
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Organizational  Pressures  and  the  Cost-Estimating 
Process:  The  Example  of  the  Early  Phases  of 
SBIRS-High 

The  ability  to  generate  realistic  and  credible  cost  estimates  during  key 
periods  in  the  history  of  our  two  cases  appears  to  have  been  nega¬ 
tively  affected  by  the  dominant  influence  exercised  by  program  offices 
over  the  cost-estimating  process  and  by  the  failure  of  senior  Air  Force 
leadership  to  insist  on  objective  cost  and  technology  risk  assessments 
because  of  budgetary  or  other  institutional  reasons. 

Many  of  the  institutional  challenges  that  we  identified  as  having 
apparently  influenced  SMC  cost  analysts  can  be  particularly  well  illus¬ 
trated  by  reviewing  the  early  phases  of  the  SBIRS-High  program,  but 
they  can  also  be  detected  at  various  other  times  during  the  history  of 
the  SBIRS-High  program  and  during  certain  phases  of  the  GPS  pro¬ 
gram.  These  circumstances  are  closely  linked  to  budgetary  politics  and 
the  competition  among  program  offices  and  among  the  services  for 
scarce  acquisition  dollars.  Program  offices  competing  for  funding  tend 
to  lose  objectivity  and  become  advocates  for  specific  systems,  as  do  the 
services  when  encouraged  by  DoD  to  form  joint  programs.  As  one  inter¬ 
viewee  summed  up  the  problem  in  a  rather  oversimplified  but  dramatic 
manner,  “Program  Managers  generally  will  not  allow  realistic  cost  esti¬ 
mates  of  their  programs  because  the  result  could  be  loss  of  budget  or 
program  termination.” 

While  it  is  difficult  to  prove  these  assertions  conclusively,  little 
doubt  exists  among  the  expert  observers  whom  we  interviewed  that 
bureaucratic,  budgetary,  and  service  politics  sometimes  undermine 
efforts  to  produce  objective,  credible  cost  estimates.  One  of  the  clearest- 
cut  and  most  well-documented  cases  took  place  during  the  early 
concept  development  phases  of  SBIRS-High.  The  early  history  of 
SBIRS-High  is  extremely  complex  and  convoluted;  what  follows  is  a 
high-level  summary  of  the  basic  outlines  to  illustrate  the  importance  of 
truly  independent  cost  estimates.17 


17  We  note  here  and  elsewhere  that,  while  organizational  independence  is  extremely  impor¬ 
tant  for  cost  analysts,  it  is  not  necessarily  sufficient  for  credible,  realistic  cost  estimates.  As  we 
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In  the  late  1980s  and  early  1990s,  several  system  concepts  both 
within  and  outside  of  the  Air  Force  were  competing  for  selection  as 
the  replacement  system  for  the  Air  Force  DSP  satellite  system.18  Selec¬ 
tion  of  an  appropriate  system  concept  and  architecture  proved  difficult 
and  contentious.  The  threat  was  complex,  and  opinions  differed  widely 
on  future  projections  of  the  threat  as  well  as  on  how  best  to  meet  it. 
Complicating  matters  further,  these  debates  took  place  while  DoD  was 
launching  its  extensive  array  of  acquisition  reform  measures. 

AWS,  FEWS,  and  ALARM 

Numerous  candidate  concepts  were  advanced  as  potential  DSP  fol¬ 
low-ons.  The  leading  contenders  included  several  candidate  concepts 
originally  advanced  by  the  Strategic  Defense  Initiative  Organization 
(SDIO,  later  the  Ballistic  Missile  Defense  Organization  or  BMDO),  an 
“official”  Air  Force  program  that  eventually  became  know  as  Follow-on 
Early  Warning  System  (FEWS),  and  an  unofficial  Air  Force  program 
called  DSP  II,  a  proposal  for  an  upgraded  version  of  the  existing  DSP. 
Disagreements  over  requirements  and  program  leadership  complicated 
selection  of  a  candidate.  This  was  particularly  true  after  the  collapse  of 
the  Soviet  Union  in  1989  reduced  the  threat  of  a  massive  nuclear  inter¬ 
continental  ballistic  missile  (ICBM)  strike.  In  addition,  the  Iraqi  inva¬ 
sion  of  Kuwait  in  1990  and  the  first  Gulf  War  raised  the  importance 
of  the  tactical  and  theater  ballistic  missile  threat  and  also  showed  that 
space-based  infrared  assets  could  be  used  for  tactical  battlefield  intel¬ 
ligence  purposes.19 

Beginning  in  1979,  the  Air  Force  began  examining  a  follow-on 
system  concept  for  the  DSP  called  the  Advanced  Warning  System 


argued  in  Chapter  Three,  independent,  high-quality  technical  risk  assessments  and  method¬ 
ologies  for  cost  analysts  are  of  at  least  equal,  and  probably  greater  importance. 

18  DSP  has  served  for  over  three  decades  as  the  main  U.S.  space-based  strategic  infrared 
detection  system  for  ballistic  missile  launches  and  nuclear  detonations. 

19  For  the  background  on  relevant  SDIO  activities  during  this  area,  see  BMDO,  Harness¬ 
ing  the  Power  of  Technology:  The  Road  to  Ballistic  Missile  Defense  from  1983-2007,  September 
2000. 
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(AWS).20  In  1984,  DoD  took  AWS  away  from  the  Air  Force  and  trans¬ 
ferred  the  responsibility  for  developing  a  DSP  mission  follow-on  system 
to  SDIO.  In  September  1987,  SDIO’s  architecture  plan  was  approved. 
It  included  the  Boost  Surveillance  and  Tracking  System  (BSTS)  in 
higher  inclined  geosynchronous  orbit  as  a  replacement  for  DSP,  and 
a  totally  new  system  called  Space-Based  Surveillance  and  Track¬ 
ing  System  (SSTS)  in  medium  orbit  (the  forerunner  of  what  became 
SBIRS-Low).  This  combination  was  intended  to  Fulfill  both  the  ballis¬ 
tic  missile  defense  missions  and  the  tactical  warning  and  attack  assess¬ 
ment  (TWAA)  missions. 

Following  the  collapse  of  the  Soviet  Union  and  the  first  Gulf  War, 
the  SDIO  architecture  was  adjusted.  Theater  Missile  Defense  (TMD) 
became  the  first  priority,  and  the  Army  and  Navy  took  the  near-term 
lead  with  various  ground  and  sea-based  anti-missile  systems.  National 
Missile  Defense  (NMD)  became  a  lower  priority,  leading  to  the  even¬ 
tual  elimination  of  the  BSTS  program  by  SDIO  and  the  restructuring 
of  some  of  the  BSTS  program  components  into  the  Brilliant  Eyes  and 
Brilliant  Pebbles  programs.  After  spending  nearly  $1  billion  on  BSTS, 
DoD  decided  to  discontinue  funding  the  program  in  April  1990. 

In  October  1990,  Congress  transferred  the  BSTS  mission  area 
back  to  the  Air  Force.  The  Air  Force  initially  focused  on  the  TWAA 
mission,  once  again  naming  the  all-new  system  AWS  as  a  reference 
back  to  1984  when  the  Air  Force  had  lost  the  original  AWS  program  to 
BMDO.  However,  although  the  Air  Force  strenuously  objected,  OSD 
soon  thereafter  recommended  terminating  AWS  because  of  potentially 
unacceptably  high  costs  and  high  technical  and  schedule  risk.  This 
decision  was  based  on  a  study  conducted  by  the  DoD  Office  of  the 
Comptroller  that  concluded  that  an  upgraded  existing  DSP  could  meet 


20  The  following  account  has  been  pieced  together  from  interviews  and  a  variety  of  pub¬ 
lished  documents,  including  General  Accounting  Office,  Early  Warning  Satellites:  Funding 
for  Follow-On  System  Is  Premature,  NSIAD-92-39,  November  1991;  General  Accounting 
Office,  Military  Space  Programs:  Comprehensive  Analysis  Needed  and  Cost  Savings  Available, 
T-NSIAD-94-164,  April  14,  1994;  U.S.  House  of  Representatives,  Committee  on  Govern¬ 
ment  Operations,  Subcommittee  on  Legislation  and  National  Security,  Strategic  Satellite 
Systems,  Testimony  of  Guido  William  Aru,  Project  Leader,  The  Aerospace  Corporation,  February 
2,  1994;  and  John  T.  Correll,  “Fogbound  in  Space,”  Air  Force  Magazine  ( 77 )  1,  January  1994. 
Also  see  BMDO,  2000. 
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most  of  the  BSTS  and  TWAA  mission  area  requirements  at  a  much 
lower  cost. 

By  April  1991,  the  Air  Force  had  responded  by  proposing  a  smaller 
and  cheaper  version  of  its  original  AWS  concept,  called  FEWS,  which 
partially  met  the  TWAA  mission  and  which  was  planned  to  exceed 
the  existing  DSP  capabilities.  One  of  the  most  important  new  techni¬ 
cal  aspects  of  FEWS  was  that  it  included  data  processing  capability  on 
board  the  space  vehicle  and  communication  cross  links.  OSD  argued 
that  this  new  system  concept  was  potentially  too  costly  and  techni¬ 
cally  risky,  recommending  at  a  minimum  that  the  onboard  processing 
requirement  be  eliminated.  This  view  was  confirmed  by  a  draft  1991 
DSB  task  force  study,  as  well  as  an  Air  Force  study  of  alternatives  con¬ 
ducted  around  the  same  time,  both  of  which  found  that  an  upgraded 
DSP  was  the  most  cost-effective  and  lowest-risk  solution. 

Throughout  this  period,  the  Air  Force  position  itself  was  far  from 
unified.  Originally  SMC  had  one  “basket  SPO”  that  included  separate 
offices  for  DSP,  FEWS,  and  Brilliant  Eyes,  each  with  a  colonel  as  pro¬ 
gram  manager.  The  DSP  program  office  continued  to  advocate  DSP  II 
to  meet  aspects  of  the  DSP  follow-on  system  missions,  the  FEWS  office 
pushed  derivatives  of  BSTS  and  Brilliant  Pebbles,  and  the  Brilliant 
Eyes  office  advocated  variants  of  the  original  SSTS.  All  three  programs 
were  competing  for  funding  and  for  mission  areas,  each  supported  by 
a  different  contractor  team.  Meanwhile,  the  Program  Executive  Office 
(PEO)  fought  to  preserve  Air  Force  control  over  the  overall  space  infra¬ 
red  surveillance  mission  area  against  perceived  encroachments  from 
the  intelligence  community  (IC).  In  this  competition  for  mission  area 
control  and  budget,  each  of  the  specific  system  proposals  and  program 
offices  had  advocacy  cost  estimates  associated  with  it,  often  driven  by 
the  contractors,  and  thus  no  unified  Air  Force  and  SMC  position  ini¬ 
tially  existed  to  counter  IC  “encroachment”  on  the  newly  reacquired 
Air  Force  mission  area. 

However,  during  1992,  FEWS  emerged  as  the  preferred  Air  Force 
system  at  the  highest  leadership  levels,  and  indeed  soon  evolved  into  the 
highest-priority  Air  Force  space  program.  By  October  1992,  AFSPC 
had  completed  a  draft  operational  requirements  document  for  FEWS, 
but  opposition  to  the  new  system  continued  at  OSD  and  elsewhere  on 
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cost  and  technology  risk  grounds,  preventing  validation  of  the  docu¬ 
ment  by  the  Joint  Requirements  Oversight  Council. 

Eventually,  Congress  requested  a  new  study  comparing  the  cost- 
effectiveness  of  FEWS  and  an  upgraded  DSR  This  led  to  an  extensive 
and  sometimes  vitriolic  debate  in  and  out  of  the  Air  Force  regarding 
the  relative  cost-effectiveness  of  various  proposed  upgraded  versions 
of  DSP  compared  with  FEWS.  This  included  allegations  presented 
formally  to  Congress  that  senior  Air  Force  echelons  had  suppressed  a 
study  sponsored  by  the  Air  Force  DSP  program  office,  the  Aerospace 
Corporation,  and  the  DSP  lead  contractors  recommending  acquisition 
of  DSP  II  rather  than  FEWS.21 

After  the  alleged  suppression  of  this  document,  the  leaking  of 
the  study  to  Congress  led  to  the  recommendation  in  June  1993  that 
the  cost-effectiveness  of  DSP  upgrades  be  once  again  compared  with 
FEWS.  This  review,  carried  out  by  the  Under  Secretary  of  Defense 
for  Acquisition  (USD(A)),  was  supported  by  a  special  technical  group 
composed  of  experts  from  several  FFRDCs.  The  technical  group  pro¬ 
duced  the  so-called  “Everett  study”  in  October  1993.  The  Everett 
study  concluded  that  future  space-based  infrared  surveillance  capabil¬ 
ity  requirements  could  be  met  by  a  much  smaller,  less  expensive,  and 
less  technologically  complex  SV  than  the  proposed  Air  Force  FEWS 
satellite  and  that  the  focus  should  shift  more  toward  the  theater  ballis¬ 
tic  missile  defense  and  battlefield  tactical  support  mission  areas.  A  key 
recommendation  of  the  Everett  study  was  that  the  SV  should  be  light 
and  small  enough  to  be  launched  from  a  medium-sized  launch  vehicle 
rather  than  the  much  more  costly  Titan  IV  heavy  launch  vehicle  that 
FEWS  would  require.  Because  of  these  findings  and  other  issues,  OSD 
cancelled  the  FEWS  program  in  late  1993. 22 

In  response  to  the  Everett  study  and  the  cancellation  of  FEWS, 
the  Air  Force  proposed  yet  another  all-new  system  in  February  1994 
to  replace  DSP  called  Alert  Eocate  and  Report  Missiles  (AEARM). 
Also  at  this  time,  the  Air  Force  moved  to  end  the  feuding  among  SMC 


21  See  U.S.  House  of  Representatives,  1994. 

22  The  requirement  for  a  smaller  SV  that  could  be  placed  in  orbit  by  a  medium-weight 
launch  vehicle  essentially  ruled  out  both  FEWS  and  DSP  II  variants  as  too  large  and  heavy. 
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program  offices  and  headed  toward  a  more  unified  position  at  SMC  to 
better  counter  what  appeared  to  some  to  be  OSD  and  IC  threats  to  its 
control  of  the  infrared  space  surveillance  mission.  To  reduce  expected 
costs  and  counter  OSD  objections,  the  proposed  initial  ALARM  sat¬ 
ellites  would  not  have  onboard  processing  and  cross-link  communi¬ 
cations  and  would  be  capable  of  launch  from  medium-lift  launchers 
through  the  use  of  new  miniaturization  technologies,  but  it  would  later 
be  upgraded  with  increased  capabilities.  However,  GAO  and  other  out¬ 
side  critics  continued  to  question  the  cost-effectiveness  and  affordabil¬ 
ity  of  ALARM,  viewing  it  essentially  as  a  downsized  and  stretched-out 
reincarnation  of  FEWS. 

OSD  Summer  Study 

The  continuing  controversy  led  to  yet  another  study  to  assess  require¬ 
ments  and  competing  system  proposals.  This  was  an  OSD  Summer 
Study  led  by  a  Senior  Steering  Group  and  sponsored  by  the  Deputy 
Assistant  Secretary  for  Intelligence  and  Security.  The  Summer  Study 
group  met  from  June  through  September  1994  and  was  under  heavy 
pressure  to  provide  a  single  definitive  answer  quickly  because  of  the 
severe  time  constraints  imposed  by  the  OSD  budget  process  and  sched¬ 
ule.  Because  of  the  demise  of  FEWS,  OSD  had  to  have  its  new  fall 
budget  position  settled  and  coordinated  by  the  end  of  the  summer. 

The  Summer  Study  ignited  a  hectic  competition  among  at  least 
three  major  design  contenders  and  institutional  players.  With  a  central 
focus  on  system  affordability,  the  competition  placed  great  pressure  on 
the  contractors,  as  well  as  on  the  cost  estimators  in  the  program  offices 
representing  each  contending  system,  to  generate  the  lowest  plausible 
cost  estimates.  Each  contender  was  represented  by  a  program  office  or 
other  government  entity  and  an  associated  contractor  team  that  viewed 
its  study  as  effectively  the  last  one  in  a  long  line  of  similar  studies,  thus 
making  it  a  “must  win”  situation.  Competing  program  offices  increas¬ 
ingly  behaved  like  advocates  for  specific  design  proposals  and  industry 
teams,  rather  than  as  objective  evaluators  of  requirements  and  design 
concepts.  The  Air  Force  PEO  followed  the  broad  Air  Force  agenda 
of  saving  ALARM,  while  SMC  program  offices  followed  their  more 
specific  agendas  driven  by  their  respective  contractors.  SDIO  and  its 
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supporting  contractors  strongly  pushed  Brilliant  Eyes.  The  IC  cham¬ 
pioned  its  own  design  and  contractor  team,  especially  its  “heritage” 
sensor  payload,  which  had  already  been  used  on  another  classified  pro¬ 
gram.  According  to  participants  in  the  Summer  Study  whom  we  inter¬ 
viewed,  the  cost  estimates  presented  by  the  competing  teams  became 
increasingly  less  credible  over  the  course  of  the  summer,  as  the  bidding 
war  escalated  along  with  claims  for  ever-increasing  capabilities  at  lower 
cost.  As  a  result,  the  Summer  Study  was  unable  to  conduct  thorough 
and  objective  cost-benefit  analyses  of  the  competing  concepts  because 
the  contractor  cost  and  technical  submissions  were  increasingly  seen  as 
lacking  in  credibility. 

Nonetheless,  the  1994  Summer  Study  successfully  determined 
the  ultimate  system  architecture  for  SBIRS-High:  four  GEO  SVs  and 
two  HEO  payloads.23  To  save  money,  it  also  recommended  use  of  the 
BSS  601  standard  commercial  satellite  bus  and,  perhaps  most  impor¬ 
tant,  called  for  use  of  modified  versions  of  the  IC  heritage  sensors  on  all 
of  the  GEO  and  HEO  satellites.  With  two  heritage  sensors  each  on  the 
four  GEO  SVs  and  one  each  for  the  HEO  payloads,  for  a  total  of  ten, 
this  use  of  an  essentially  common  sensor  package  for  all  satellites  was 
expected  to  result  in  much  lower  unit  procurement  costs.  In  addition, 
the  recommended  reuse  of  existing  software  from  the  heritage  sensor 
was  also  intended  to  reduce  costs.  Since  the  IC  had  presented  actual 
cost  numbers  for  the  heritage  sensor  based  on  its  use  on  a  classified  pro¬ 
gram,  the  Summer  Study  participants  were  reasonably  confident  about 
the  cost  estimates  for  at  least  this  one  critical  area. 

The  Summer  Study  recommended  that  all  SDIO  active  missile 
defense  requirements  be  moved  to  the  Brilliant  Eyes  follow-on  (later 
SBIRS-Low)  and  that  procurement  of  this  system  be  deferred  pend¬ 
ing  a  final  decision  in  FY00.  The  Summer  Study  also  recommended 
the  establishment  of  a  joint  program  office  led  by  the  Air  Force  but 
including  the  IC  to  facilitate  provision  of  the  heritage  sensor.  Finally, 
the  Summer  Study  assumed  that  a  streamlined  acquisition  approach 
similar  to  that  used  by  the  IC  would  be  adopted  to  save  costs.  The  cost 


23  The  HEO  sensor  payloads  would  be  hosted  on  a  classified  satellite. 
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estimates  produced  by  the  Summer  Study  were  based  on  all  the  above 
assumptions. 

Senior  Air  Force  leaders  were  allegedly  not  happy  with  the  find¬ 
ings  of  the  Summer  Study,  because  they  essentially  endorsed  the  IC 
design  concept  and  architecture  instead  of  ALARM  and  called  for  use 
of  the  IC  heritage  sensor  payload  for  all  satellites.  According  to  some 
Summer  Study  participants  and  other  officials,  senior-level  Air  Force 
leaders  strongly  opposed  granting  the  IC  a  significant  role  in  designing, 
developing,  and  managing  a  future  system,  which  was  considered  to  be 
rightfully  an  Air  Force-dominated  effort. 

In  the  wake  of  the  Summer  Study,  the  Air  Force  brought  in  a  new 
program  manager  to  SMC  in  early  1995  to  end  the  feuding  among  the 
different  program  offices  in  the  SMC  basket  SPO  and  to  help  build  a 
more  unified  Air  Force  position.  Two  of  the  three  program  directors 
were  reassigned,  and  FEWS,  DSP,  and  Brilliant  Eyes  were  consolidated 
into  a  single  program  office  under  the  new  program  manager.  The  new 
system  program  office  was  named  the  Space  Based  Infrared  System. 
Now  all  requirements  were  to  be  satisfied  by  a  single  program.  The  Air 
Force  then  funded  a  pre-EMD  design  concept  competition  between 
two  contractor  teams.  One  included  the  contractor  that  had  developed 
and  integrated  the  DSP  SV,  teamed  with  the  heritage  sensor  contractor 
originally  advocated  by  the  IC  and  effectively  endorsed  by  the  OSD 
Summer  Study.  The  other  team  was  led  by  one  of  the  former  FEWS 
contractors  teamed  with  the  DSP  payload  contractor  and  included  only 
a  paper  design  for  an  all-new  sensor  payload.24  During  the  course  of 


24  Some  Air  Force  interviewees  argued  that  the  difference  in  the  relative  maturity  of  the  two 
competing  sensor  designs  is  often  exaggerated.  They  claim  that  the  heritage  sensor,  as  used 
in  the  new  proposal  from  the  DSP  contractor  team,  required  considerable  modification, 
and  that  the  new  paper  design  for  a  sensor  offered  by  the  other  contractor,  benefited  from 
earlier  hardware  experience  on  the  DSP  sensor  and  Defense  Meteorological  Satellite  Pro¬ 
gram  (DMSP).  Some  interviewees  pointed  out  that  the  second  prime  contractor  (Lockheed 
Martin)  at  one  point  had  approached  the  heritage  sensor  contractor  to  form  a  second  team 
to  develop  their  own  design  that  would  also  have  used  a  modified  version  of  the  heritage 
sensor.  This  indeed  was  allegedly  the  preferred  solution  of  the  senior  civilian  leadership  in 
SAF/AQ — that  is,  both  primes  teaming  with  the  heritage  sensor  subcontractor.  However,  it 
is  alleged  that  the  heritage  sensor  contractor  declined  to  team  with  Lockheed  Martin,  prefer¬ 
ring  to  remain  in  its  existing  team  for  business  reasons. 
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this  competition,  the  Air  Force  moved  further  away  from  the  Summer 
Study  findings  and  closer  to  the  earlier  preferred  all-Air  Force  ALARM 
concepts.  For  example,  the  original  ALARM  long-term  requirement 
for  SV  onboard  data  processing  migrated  back  into  the  program.25 

Nonetheless,  the  competition  between  the  two  contractors,  along 
with  major  budgetary  constraints,  pressured  both  contractors  to  keep 
their  cost  estimates  as  close  as  possible  to  those  that  had  emerged  from 
the  1994  OSD  Summer  Study,  even  though  those  estimates  were  based 
on  key  assumptions  regarding  a  variety  of  technical  issues  that  were  no 
longer  necessarily  valid,  such  as  no  onboard  processing  and  use  of  a 
common  modified  heritage  sensor  payload  on  all  SVs. 

In  late  1996,  the  Air  Force  completed  its  down-select  process; 
Lockheed  Martin,  a  former  lead  contractor  on  FEWS  and  ALARM, 
was  the  winner.  Many  officials  involved  with  the  program  at  the  time 
believed  that  the  winning  contractor’s  program  cost  estimates,  as  well  as 
the  official  program  office  estimates,  which  were  roughly  similar  to  the 
Summer  Study  estimates  based  on  completely  different  assumptions, 
were  extremely  optimistic  and  that  the  overall  program  was  seriously 
underfunded.  This  was  particularly  true  since  the  Lockheed  Martin 
proposal  did  not  include  the  modified  heritage  sensor  and  software,  but 
rather  had  an  all  new  scanner  and  starer  for  each  GEO  satellite  and  a 
scanner  for  the  host  FIEO  satellites. 

This  brief  and  admittedly  incomplete  recounting  of  the  early  pre¬ 
development  stages  of  SBIRS-High  illustrates  some  of  the  problems 
that  occur  when  the  cost-estimating  process  is  controlled  or  heavily 
influenced  by  major  program  stakeholders  that  have  become  advo¬ 
cates  for  specific  systems  and  contractor  design  teams.  The  lack  of  a 
truly  independent  cost-estimating  capability  with  a  strong  independent 
voice  inhibited  the  objective  cost-benefit  analysis  within  the  Air  Force 
of  competing  Air  Force  systems  (DSP  II,  FEWS,  ALARM,  Brilliant 
Eyes).  In  addition,  the  lack  of  independence  of  the  cost-estimating  pro¬ 
cess  and  the  institutional  and  bureaucratic  parochialism  exhibited  by 
all  participants  in  the  Summer  Study,  including  the  Air  Force,  the  IC, 


25  One  interviewee  argued  that  this  requirement  was  restored  mainly  because  the  Army, 
which  would  be  an  important  user,  strongly  supported  it. 
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and  the  SDIO,  all  undermined  the  ability  of  the  1994  OSD  Summer 
Study  participants  to  conduct  fully  transparent,  credible,  objective 
cost-benefit  analyses  of  the  various  options  to  fulfill  the  space-based 
infrared  surveillance  mission.  Of  course,  this  situation  was  hardly  the 
fault  of  the  Air  Force  alone.  For  example,  the  IC  constantly  touted  its 
lower-cost  figures  for  its  heritage  sensor  but  refused  to  share  its  cost 
methodology,  databases,  or  costing  assumptions  with  the  Air  Force, 
OSD  officials,  or  even  with  the  Summer  Study  participants.26 

This  less-than-optimal  situation  was  not  unique  to  the  pre-EMD 
period  of  SBIRS-FIigh,  or  indeed  to  SBIRS-FIigh  itself.  For  example, 
during  2002  when  SBIRS-FIigh  was  being  re-baselined  due  to  Nunn- 
McCurdy  breaches27  and  had  to  be  recertified  to  continue,  significant 
pressures  were  apparently  felt  by  the  SPO  cost  estimators  to  keep  the 
estimates  fairly  optimistic  because  the  program  was  clearly  at  risk  of 
being  cancelled.  Outside,  independent  observers  at  the  time  argued 
that  the  estimates  produced  by  the  SPO  during  the  re-baselining  were 
unjustifiably  optimistic  because  of  remaining  unresolved  technology 
risks.28  It  appears  that  the  continuing  lack  of  independence  in  the  cost¬ 
estimating  process  was  a  major  contributing  factor  to  this  optimism. 

Nor  was  our  other  case  study,  GPS,  immune  to  similar  problems, 
although  they  were  on  a  much  smaller  scale.  It  appears  that  when  GPS 


26  Some  Air  Force  interviewees  argued  that  the  true  root  cause  of  SBIRS-High  cost  growth 
problems  should  be  sought  not  in  the  estimating  process  but  in  the  budgeting  process.  They 
point  out  that  the  official  Air  Force  estimate  at  the  beginning  of  the  program  was  higher  than 
the  contractor’s  estimate  but  that  the  original  official  budget  was  lower  than  both  the  Air 
Force  and  even  the  contractor’s  optimistic  estimate.  They  insist  that  the  unrealistic  budget 
forced  the  contractor  to  cut  discretionary  but  crucial  areas  during  the  first  three  years  of  the 
program,  such  as  system  engineering,  quality  assurance,  and  vendor  surveillance.  They  argue 
that  these  were  the  root  causes  of  the  technical  problems.  However,  others  maintain  that  it 
is  highly  likely  that  the  program  would  still  have  encountered  significant  technical  problems 
and  cost  growth,  even  if  the  budget  had  covered  the  full  original  official  Air  Force  estimate, 
given  the  enormous  cost  growth  experienced  by  the  program  over  its  history.  The  original 
official  Air  Force  estimate  was  only  tens  of  percent  different  from  the  contractor  estimate. 
Yet  cost  growth  to  date  has  been  well  in  excess  of  200  percent. 

27  As  noted  earlier,  a  “Nunn-McCurdy”  unit  cost  breach  occurs  when  a  MDAP  has  an 
increase  of  at  least  15  percent  in  program  acquisition  unit  cost. 

28  As  reported  in  Bonds  et  ah,  2003. 
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was  extensively  restructured  in  1999-2000  to  include  modernization  of 
the  IIF  and  IIR-M  satellites,  SPO  cost  estimators  developed  overly  opti¬ 
mistic  estimates,  particularly  for  GPS  IIF  and  for  OCS.  Although  it  is 
more  difficult  in  this  case  to  demonstrate  with  certitude,  it  seems  likely 
that  funding  constraints  and  the  preference  of  the  SPO  leadership  to 
move  the  program  forward  without  a  major  program  milestone  review 
influenced  the  cost  estimates  and  contributed  to  its  optimism.29 

As  we  have  already  noted,  however,  institutional  independence 
does  not  necessarily  guarantee  high-quality,  credible  cost  estimates.  For 
example,  in  2002,  a  new  official  SBIRS-FIigh  program  cost  estimate, 
independent  of  both  the  program  office  and  the  Air  Force,  eventually 
became  the  new  baseline  for  the  program.  This  estimate  was  developed 
by  the  OSD  CAIG.  Although  the  estimate  was  higher  than  the  SPO 
estimate,  even  this  baseline  had  been  severely  breached  by  2004-2005, 
requiring  a  new  Nunn-McCurdy  recertification  process.  Therefore,  it 
could  be  argued  that  the  independence  of  the  cost-estimating  process 
produced  only  modestly  better  results  than  those  produced  by  the  SPO. 
We  do  not  necessarily  disagree  with  this  conclusion.  While  we  argue 
that  independence  is  an  important  factor,  it  is  not  a  sufficient  condi¬ 
tion  for  producing  credible  cost  estimates  in  and  of  itself.  Perhaps  most 
important,  accurate  assessment  of  technology  risk  and  maturity  must 
be  incorporated  into  cost  estimates  in  a  more  effective  manner,  as  we 
discussed  at  length  in  Chapter  Three. 

We  argue  below  that  the  government  acquisition  authorities  must 
also  conduct  rigorous  oversight  and  assessments  of  contractor  propos¬ 
als  before  contract  awards,  and  of  contractor  progress  after  contract 
awards,  to  generate  and  maintain  credible  cost  estimates.  Finally,  and 
perhaps  most  important,  as  we  argued  in  Chapter  Three,  an  indepen¬ 
dent,  objective,  and  well-informed  technical  risk-assessment  process 
directly  supporting  the  cost-estimating  process  is  critical  for  the  gen¬ 
eration  of  credible  cost  estimates. 


29  Again,  the  important  issue  of  credible  and  independent  technical  risk  assessment  as  it 
relates  to  the  GPS  program  is  discussed  in  Chapter  Three. 
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The  Context:  A  Changing  Defense  Industrial  Base 
in  the  1990s 

In  this  section,  we  conclude  by  once  again  summarizing  and  briefly 
reviewing  the  broader  context  regarding  the  industrial  base  environ¬ 
ment  in  which  these  programs  were  launched  and  that  played  a  role  in 
increasing  the  challenges  facing  cost  estimators  and  other  program  offi¬ 
cials,  as  well  as  contractors.  These  contextual  factors  greatly  complicated 
the  challenges  posed  to  credible  cost  estimating  by  the  new  emphasis 
on  TSPR  and  acquisition  reform  that  arose  in  the  early  1990s. 

Two  contextual  factors  are  particularly  relevant:  (1)  industry 
downsizing  and  smaller  numbers  of  large  programs  leading  to  increased 
competition  among  the  remaining  prime  contractors,  which  tempted 
contractors  to  overpromise  with  unrealistic  bids  at  a  time  when  their 
capabilities  to  deliver  were  declining  and  the  ability  of  SPOs  to  rec¬ 
ognize  unrealistic  bids  was  decreasing;  and  (2)  the  rapidly  increas¬ 
ing  inherent  technological  complexity  and  difficulty  of  military  space 
development  programs.  The  point  is  to  reemphasize  that  even  if  cred¬ 
ible  cost  estimating  on  the  SBIRS-High  and  GPS  programs  had  not 
been  negatively  affected  by  the  acquisition  reform  policies  and  organi¬ 
zational  and  institutional  issues  discussed  above,  it  would  have  faced 
serious  challenges  due  to  the  unique  historical  industrial  base  and 
technological  background  environments  in  which  the  programs  were 
launched. 

Industry  Downsizing  and  Growing  Competition  for  Fewer  Programs 

The  decade  of  the  1990s  was  a  period  of  declining  defense  budgets, 
which  led  to  a  further  decline  in  new  program  starts  and  downsiz¬ 
ing  and  consolidation  of  the  aerospace  defense  industrial  base.  These 
factors  strongly  encouraged  acquisition  reform  as  a  means  to  reduce 
system  procurement  and  operating  costs  and  also  spurred  DoD  acqui¬ 
sition  workforce  reductions.  Defense  industry  employment  peaked 
in  1987  and  fell  thereafter.  This  decline  rapidly  accelerated  with  the 
fall  of  the  Berlin  Wall  in  1989  and  the  onset  of  a  recession  in  1990. 
Hundreds  of  thousands  of  aerospace  jobs  disappeared  in  the  early 
1990s.  Fewer  students  entered  aerospace  engineering  programs,  and 
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foreign  competition  for  commercial  launch  services  and  commer¬ 
cial  satellite  programs  appeared  to  be  growing  dramatically.  The 
U.S.  space  industry  was  forced  to  downsize  and  consolidate.  Accord¬ 
ing  to  one  accounting,  the  number  of  major  space  companies  in  the 
United  States  declined  from  20  to  three  (Boeing,  Lockheed  Martin, 
and  Raytheon)  from  the  mid-1980s  through  1997.30  One  of  the  fore¬ 
most  military  satellite  contractor  sites  lost  approximately  two-thirds  of 
its  engineering  and  technical  staff  during  this  period.  With  increas¬ 
ing  frequency,  experts  voiced  concerns  about  the  declining  skill  base 
of  the  space  industry  and  the  long-term  ability  of  the  U.S.  aerospace 
industry  to  maintain  a  global  leadership  in  this  new  environment.31 

Downsizing  and  consolidation,  which  were  taking  place  through¬ 
out  the  1990s,  could  often  be  highly  disruptive  for  ongoing  programs, 
including  our  case  studies.  A  good  example  is  the  numerous  changes 
that  affected  the  prime  contractor  developing  the  GPS  IIF  SV  during 
this  period.  The  prime  contractor  for  the  GPS  IIF  was  Boeing  Space 
and  Intelligence  Systems  (S&IS)  headquartered  in  Seal  Beach,  Califor¬ 
nia.  However,  throughout  the  IIF  contract,  the  prime  contractor  expe¬ 
rienced  considerable  turbulence  due  to  mergers  and  industry  restruc¬ 
turing,  leading  to  numerous  changes  in  the  designated  development 
and  manufacturing  sites,  as  well  as  issues  regarding  clashes  in  corporate 
cultures. 

GPS  government  program  officials  believe  that  this  turbulence 
negatively  influenced  the  development  and  production  program.  In 
1996,  the  Air  Force  awarded  the  original  IIF  contract  to  North  Amer¬ 
ican  Rockwell  Aerospace  in  Seal  Beach  and  Anaheim,  California, 
the  original  developer  of  the  GPS  Blocks  I  and  II.  In  late  1997,  the 
Boeing  Company  bought  Rockwell  and  acquired  the  Rockwell  Aero¬ 
space  Seal  Beach  facility  and,  with  it,  the  GPS  IIF  program.  In  2000, 
Boeing  purchased  the  Hughes  satellite  manufacturing  operations,  for- 


30  Michel  Andrieu  and  Pierre-Alain  Schieb,  “Space:  The  Forgotten  Frontier?”  OECD 
Observer ,  April  2003. 

31  Typical  expressions  of  those  concerns  can  be  found  in  Vice  President’s  Space  Policy  Advi¬ 
sory  Board,  The  Future  of  the  US.  Space  Industrial  Base:  A  Task  Group  Report ,  Washington 
D.C.,  November  1992. 
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merly  Hughes  Space  and  Communications  Company,  in  El  Segundo, 
California,  for  $3.75  billion,  and  renamed  it  Boeing  Satellite  Systems. 
During  this  period,  Boeing  acquired  McDonnell  Douglas,  as  well  as 
Rocketdyne  Aerospace  facilities  in  Canoga  Park. 

Boeing  spent  several  years  reorganizing  and  shifting  development 
and  production  work  among  its  newly  acquired  space  and  information 
technology  facilities  in  Southern  California.  Development  work  on  the 
GPS  IIF  was  moved  from  Seal  Beach  to  Anaheim,  then  back  to  Seal 
Beach,  and  finally  to  Huntington  Beach.  SV  production  was  moved 
from  Seal  Beach  to  El  Segundo.  In  addition,  integrating  the  corporate 
cultures  of  the  former  Hughes  El  Segundo  engineers  with  the  new  Seal 
Beach  headquarters  created  its  own  set  of  challenges.  According  to  GPS 
government  program  officials,  the  frequent  changes,  reorganizations, 
and  problems  with  integrating  formerly  separate  corporate  cultures  led 
to  significant  challenges  for  the  program  and  have  likely  contributed  to 
technical  problems,  cost  growth,  and  schedule  delays. 

Declining  defense  budgets  in  the  1990s  also  meant  fewer  major 
programs.  As  industry  consolidation  continued,  the  remaining  aero¬ 
space  giants  became  locked  in  fierce  competitions  to  win  the  shrinking 
number  of  major  new  programs.  Imbued  with  the  principles  of  acqui¬ 
sition  reform,  the  government  managers  of  new  space  R&D  efforts 
focused  enormous  attention  on  attempting  to  reduce  procurement  and 
life  cycle  costs  through  greater  use  of  COTS  technology  and  parts  and 
the  transfer  of  more  program  responsibility  to  contractors.  The  surviv¬ 
ing  contractors  heard  the  government’s  message  that  they  had  to  provide 
better  products,  at  less  cost,  with  fewer  human  and  dollar  resources, 
and  with  much  less  government  supervision  and  oversight.  Desperate  to 
win  the  relatively  few  remaining  new  large-scale  contracts,  companies 
gladly  promised  to  satisfy  the  government’s  requirements  at  the  lowest 
possible  costs — while  not  always  being  certain  how  this  could  be  done. 
Intense  competitive  pressures  and  a  very  strong  emphasis  on  reducing 
costs,  combined  with  the  new  relaxed  environment  of  minimal  gov¬ 
ernment  oversight  and  contractor  reporting  requirements  promoted  by 
acquisition  reform  measures,  were  bound  to  encourage  unusually  opti¬ 
mistic  contractor  cost  estimates  and  technical  risk  assessments  at  a  time 
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when  the  government  acquisition  community  was  reducing  its  ability 
to  monitor  and  validate  contractor  estimates  and  assessments.32 

These  trends  all  coalesced  at  a  time  when  military  space  sys¬ 
tems  were  becoming  dramatically  more  technologically  sophisticated 
and  complex.  The  databases  used  in  traditional  space  cost-estimating 
models  became  increasingly  obsolete  in  this  environment,  and  acqui¬ 
sition  reform  measures  eliminated  much  of  the  routine  collection  of 
new  cost  and  technical  data  that  had  been  common  in  the  past.  The 
increased  technological  complexity  of  proposed  new  military  space 
systems  challenged  the  technical  and  system  engineering  knowledge 
of  the  government  acquisition  officials  tasked  with  the  job  of  assess¬ 
ing  proposals  and  contractor  progress — at  a  time  when  the  acquisition 
workforce  was  experiencing  significant  downsizing. 

Increased  Complexity  of  Space  Systems 

A  variety  of  factors  coalesced  in  the  1990s,  leading  to  the  emergence 
of  much  more  complex  space  “systems  of  systems.”  The  system-of- 
systems  concept  gained  wide  favor  beginning  in  the  1990s  as  a  key 
component  of  the  “Revolution  in  Military  Affairs.”33  A  system  of  sys¬ 
tems  is  composed  of  multiple  separate  systems,  subsystems,  sensors, 
and  other  components  all  netted  together  into  a  single  complex  inter¬ 
active  system. 34According  to  one  expert  definition,  systems  of  systems 


32  A  much  more  detailed  discussion  of  acquisition  reform  and  its  effects  on  the  case  study 
programs  follows  below. 

33  For  example,  see  Admiral  William  A.  Owens,  Vice  Chairman,  Joint  Chiefs  of  Staff,  The 
Emerging  U.S.  System-of-Systems,  National  Defense  University  Strategic  Forum,  No.  63,  Feb¬ 
ruary  1996. 

34  A  basic  definition  from  the  period  states:  “Systems  of  systems  are  large  scale  concurrent 
and  distributed  systems  that  are  comprised  of  complex  systems.”  See  V.  Kotov,  Systems  of 
Systems  as  Communicating  Structures,  Hewlett  Packard  Computer  Systems  Laboratory  Paper, 
HPL-97-124,  1997,  p.  1-15.  Another  emphasizes  information  superiority:  “In  relation  to  joint 
warfighting,  system  of  systems  is  concerned  with  interoperability  and  synergism  of  Com¬ 
mand,  Control,  Computers,  Communications,  and  Information  (C4I)  and  Intelligence, 
Surveillance,  and  Reconnaissance  (ISR)  Systems.”  See  W.  H.  Manthorpe,  “The  Emerging 
Joint  System  of  Systems:  A  Systems  Engineering  Challenge  and  Opportunity  for  API  f Johns 
Hopkins  APT  Technical  Digest  17(3),  1996,  p.  55-60. 
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are  characterized  by  “complex  combinations  and  interdependencies  of 
technologies,  operations,  tactics,  and  procedures.”35 

The  much-increased  complexity  of  military  space  systems  netted 
together  into  larger  systems  of  systems  posed  many  new  challenges  for 
the  space  acquisition  and  cost-estimating  communities.  The  so-called 
Young  Report,  a  widely  influential  joint  Air  Force/DoD  study  pub¬ 
lished  in  2003  on  the  growing  challenges  in  the  space  acquisition  pro¬ 
cess,  pinpointed  the  system-of-systems  concept  involving  multiple  users 
and  extensive  user  requirements  as  a  major  contributor  to  the  causes  of 
cost  growth  on  military  space  systems.  It  noted  that  the  proliferation 
of  users  and  requirements  led  to  increasingly  complex  systems  of  sys¬ 
tems,  which  greatly  increased  the  difficulty  of  managing  cost,  schedule, 
and  risk.36  At  the  same  time,  the  complexity  of  individual  systems, 
subsystems  and  technologies  was  also  increasing,  as  sensors  and  other 
payloads  became  technologically  more  sophisticated,  and  much  more 
complex  processing  and  software  tasks  migrated  to  space  vehicles. 

These  trends  toward  greater  system  and  “system  of  systems”  com¬ 
plexity  arose  at  the  same  time  that  the  government  was  transferring 
more  system  engineering  and  design  responsibility  to  the  contractor, 
and  reducing  government  oversight  and  guidance,  in  accordance  with 
acquisition  reform  concepts  and  measures. 

Adding  to  the  challenges  for  cost  analysts  was  DoD’s  dramatically 
increased  emphasis  placed  on  using  COTS  technology,  parts,  subsys¬ 
tems,  and  software  on  satellites  to  reduce  costs  and  shorten  schedules. 
Greater  use  of  COTS  was  a  central  theme  of  DoD’s  acquisition  reform 
initiatives  in  the  1990s.  As  noted  elsewhere,  later  experience  suggests 
that  the  use  and  integration  of  COTS  hardware  and  parts,  and  espe- 


35  Dennis  J.  Anderson,  James  E.  Campbell,  and  Leon  D.  Chapman,  Evaluating  A  Complex 
System  of  Systems  Using  State  Modeling  and  Simulation,  presentation  at  the  National  Defense 
Industrial  Association  Systems  Engineering  Conference,  San  Diego,  Calif.,  October  20—23, 
2003. 

36  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics, 
Report  of  the  Defense  Science  Board/Air  Force  Scientific  Advisory  Board  Joint  Task  Force  on 
Acquisition  of  National  Security  Space  Programs,  Washington,  D.C.,  May  2003,  p.  2. 


98  Improving  the  Cost  Estimation  of  Space  Systems 


dally  software,  proved  far  more  technologically  challenging  than  origi¬ 
nally  anticipated  by  the  acquisition  reform  theorists.37 

The  two  contextual  trend  areas  briefly  touched  on  above — indus¬ 
try  downsizing,  reduced  numbers  of  programs,  and  increased  com¬ 
petition;  and  increased  space  system  and  technological  complexity — 
significantly  affected  the  space  defense  industrial  base  environment  in 
the  1990s,  complicating  the  challenges  confronting  industry  as  well 
as  government  cost  analysts  and  other  acquisition  professionals.  Com¬ 
bined  with  the  emergence  of  TSPR  and  the  increasing  popularity  of 
new  acquisition  reform  measures  strongly  advocated  by  the  Clinton 
administration,  these  contextual  factors  contributed  to  the  difficulties 
encountered  by  the  cost  analysis  community  in  the  mid  1990s  when 
SBIRS-High  and  GPS  IIF  were  entering  development. 


Summary  Observations 

In  summary,  a  careful  review  of  the  history  of  the  cost-estimating  pro¬ 
cesses  in  the  SBIRS-High  and  GPS  programs  suggests  the  following: 

•  Organizational  and  bureaucratic  independence  from  the  program 
office  is  an  extremely  important,  though  not  sufficient,  condition 
for  ensuring  the  generation  of  credible  cost  estimates. 

•  Rigorous  oversight,  monitoring,  and  assessment  of  contractor 
costs  and  cost  data,  throughout  all  phases  of  the  proposal  process 
and  continuously  throughout  program  execution,  are  critical  for 
the  development  of  credible  cost  estimates. 

Finally,  our  case  studies  demonstrate  that  the  single  biggest  chal¬ 
lenge  for  the  cost-estimating  process  has  been  the  development  of  cred¬ 
ible  methodologies  and  approaches  for  determining  technology  risk, 
quantifying  technical  risk,  and  incorporating  the  technical  risk  assess- 


37  For  a  detailed  discussion  of  some  of  these  challenges,  see  Lorell  et  al.,  2000,  and  Chapter 
Three  of  this  monograph. 
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ment  into  the  cost-estimating  process  in  order  to  produce  a  credible 
final  cost  estimate,  as  discussed  at  length  in  Chapter  Three. 

The  next  chapter  examines  the  current  organization  and  work¬ 
load  of  the  SMC  financial  organization  to  see  what  needs  to  be  done  so 
that  future  programs  can  reduce  the  types  of  challenges  encountered 
by  the  SBIRS-High  and  GPS  programs  in  the  past. 


CHAPTER  FIVE 


SMC  Organization  and  Cost  Analysis  Workforce 
Assessment 


This  chapter  contains  three  parts.  It  begins  with  a  description  of  the 
Space  and  Missile  Systems  Center  organization,  continues  with  a  more 
detailed  discussion  of  the  SMC  Financial  Management  (SMC/FM) 
organization,  and  concludes  with  an  analysis  of  the  supply  and  demand 
for  cost-analyst  staffing  at  SMC.  The  analysis  is  a  snapshot  of  current 
staffing;  note  that  we  do  not  make  any  judgment  on  the  quality  of  the 
workforce. 


Description  of  Space  and  Missile  Systems  Center 

This  section  describes  SMC,  providing  some  details  about  its  history, 
organization,  and  staffing  and  distinguishing  among  civil  service,  mili¬ 
tary,  and  contractor  personnel.  It  also  describes  the  staffing  of  similar 
cost-estimating  organizations. 

History 

What  is  now  the  Air  Force  Space  and  Missile  Systems  Center  began 
in  the  mid-1950s  as  the  Western  Development  Division  of  the  Air 
Research  and  Development  Command.  It  was  soon  renamed  the  Air 
Force  Ballistic  Missile  Division  (AFBMD),  reflecting  its  primary  mis¬ 
sion  of  developing  strategic  missiles.  By  1961,  AFBMD  had  assumed 
responsibility  for  development  of  most  military  space  systems  and  the 
boosters  to  launch  them.  Over  the  next  30  years,  the  organization  went 
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through  a  succession  of  name  changes,  as  the  responsibilities  for  stra¬ 
tegic  missile  development  were  first  moved  to  a  separate  command  and 
later  restored.  In  1992,  the  functions  were  recombined  for  the  last  time, 
and  the  command  was  renamed  Space  and  Missile  Systems  Center. 

As  a  result  of  the  desire  to  foster  better  integration  between  the 
developers  and  users  of  space  systems,  in  2001  SMC  was  transferred 
from  Air  Force  Materiel  Command  to  become  the  acquisition  arm  of 
Air  Force  Space  Command  (AFSPC).  AFSPC  also  has  responsibility 
for  space  launch  and  flight  operations.  Since  2002,  the  SMC  com¬ 
mander  has  assumed  the  added  responsibilities  of  program  executive 
officer  for  space,  reporting  to  the  space  acquisition  executive. 

Organization 

The  Space  and  Missile  Systems  Center  recently  reorganized  its  approxi¬ 
mately  6,800-person  workforce  to  parallel  the  wing/group/squadron 
structure  of  an  Air  Force  operational  command.  SMC  is  now  struc¬ 
tured  into  six  wings,  21  groups,  12  squadrons,  20  divisions,  two  system 
offices,  and  the  61st  Air  Base  Wing.  As  shown  in  Figure  5.1,  these  orga¬ 
nizations  roughly  parallel  the  previous  system  program  directorates, 
program  offices,  and  staff  functions.  General  descriptions  of  the  major 
SMC  programs  appear  in  Appendix  B.  The  intent  of  the  reorganiza¬ 
tion,  which  has  also  been  implemented  at  other  Air  Force  acquisition 
centers,  is  to  “.  .  .  provide  respective  commanders  with  strengthened 
authority,  accountability,  and  responsibility.”1 

While  in  some  respects  this  reorganization  simply  renames  the 
existing  elements,  which  retain  their  previous  personnel  and  functions, 
its  advertised  objective  of  giving  the  unit  commander  increased  author¬ 
ity  and  flexibility  over  the  organization  may  have  significant  effects  on 
the  more  specialized  functions  such  as  cost  analysis.  It  appears  that 
in  the  new  organization  each  commander  will  have  nearly  complete 
responsibility  for  setting  staffing  levels  and  skill  mix,  as  well  as  evalu¬ 
ating  all  personnel  assigned  to  his  or  her  organization.  Although  this 
approach  has  advantages  for  the  mainstream  functions  of  planning  and 


1  Air  Force  Space  and  Missile  Center,  Office  of  Public  Affairs,  Press  Release  No.  06-07-06, 
2006. 


Figure  5.1 
SMC  Organization 
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executing  the  program,  it  may  also  have  the  unintended  consequence 
of  effectively  isolating  assigned  cost  analysts  within  the  organizational 
stovepipe  of  the  program,  limiting  training,  career  development,  cross¬ 
fertilization,  and  functional  independence.  The  cost  analysis  functional 
staff  office  (SMC/FMC)  is  located  in  the  Financial  Management  and 
Comptroller’s  office,  which  reports  to  the  SMC  commander. 


Staffing  of  the  Cost  Function 

Up  until  the  mid-1990s,  the  cost  analysts  at  SMC  were  assigned  either 
to  the  Comptroller’s  staff  (SMC/FMC)  and  or  to  one  of  the  system  pro¬ 
gram  offices.  The  SMC/FMC  staff  generally  performed  independent 
cost  estimates  (ICE)  and  analyses,  directed  cost  research  to  improve 
cost  tools  and  methodologies,  assisted  programs  offices  as  needed, 
and  set  cost  analysis  policies  for  the  Center.  For  a  variety  of  reasons, 
cost  analysis  activities  and  experienced  personnel  gradually  migrated 
from  the  staff  to  the  program  offices.  This  process  culminated  with  the 
complete  dissolution  of  the  FMC  organization  as  a  directorate  in  the 
mid-1990s  and  reassignment  of  its  responsibilities  to  other  staff  organi¬ 
zations  or  to  the  program  offices.  In  parallel  with  the  reduced  empha¬ 
sis  on  USAF  cost  analysis,  general  reductions  across  the  acquisition 
workforce  resulted  in  a  reduced  capability  on  the  part  of  the  USAF  to 
effectively  manage  complex  acquisition  programs.  In  2005,  the  SMC 
commander  reestablished  the  cost  analysis  organization  as  a  directorate 
within  the  Comptroller’s  staff. 

At  SMC,  government  cost  analysis  positions  are  staffed  by  a  com¬ 
bination  of  civil  service  personnel,  military  officers,  and  SETA  contrac¬ 
tors.  We  briefly  discuss  each  personnel  category  below. 

Civil  Service 

In  the  Air  Force,  the  cost  analysis  function  is  considered  part  of  the 
financial  management  (FM)  career  field,  which  also  encompasses 
budget  and  accounting  and  finance  specialties.  Over  the  course  of  a 
civil  service  career,  most  Air  Force  FM  personnel  will  be  assigned  to 
at  least  two  of  these  functional  areas  at  various  times.  While  finan- 
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cial  management  skills  and  experience  are  no  doubt  useful  for  cost 
analysts,  the  more  specialized  skills  and  experience  requirements  for 
cost  analysis  are  not  necessarily  interchangeable  with  those  of  the  other 
financial  management  specialties.  For  example,  cost  analysis  typically 
requires  a  higher  level  of  quantitative  skills  (primarily  probability  and 
statistics)  and  less  accounting  than  most  other  FM  positions.  Also,  cost 
analysts  in  acquisition  commands  must  often  deal  with  technical  issues 
in  some  depth  to  ensure  that  technology  risk  is  appropriately  reflected 
in  their  estimates.  Similarly,  experience  with  a  variety  of  programs  in 
different  phases  of  the  acquisition  cycle  is  important  for  a  cost  analyst 
to  develop  the  perspective  needed  to  identify  and  assess  cost,  technical, 
and  programmatic  risk  in  new  programs.  The  box  below  provides  an 
example  of  the  criteria  for  mid-level  acquisition  cost  estimators  at  the 
National  Reconnaissance  Office  (NRO). 


Skills 

In-depth  knowledge  of  principles,  theories,  and  methods  of  cost  analysis, 
including  developing  cost-estimating  relationships  and  designing  cost-estimating 
methods  and  models,  economic  analysis,  and  statistical  analysis 

Independent  assessments  of  cost,  technical  and  schedule  baselines,  identifying 
risk,  and  potential  cost  effects 

Strong  knowledge  of  mathematics,  economics,  finance,  and  statistics 

Knowledge  and  understanding  of  key  technical  and  performance  characteristics 
of  complex  satellite  systems  as  they  relate  to  cost 

Strong  computer  skills,  especially  in  statistics,  spreadsheets,  and  presentation 
applications 

Experience 

Four  years  of  progressively  responsible  acquisition  cost  analysis  experience  (two 
years  with  master's  degree  in  operations  research,  cost  analysis,  finance,  business 
administration,  engineering,  physics,  mathematics  or  computer  science,  or  other 
quantitative  analysis  field) 

Education 

Bachelor's  degree  in  engineering,  physics,  operations  research,  cost 
analysis,  mathematics,  or  master's  degree  in  finance,  economics,  or  business 
administration  with  at  least  20  credit  hours  in  a  quantitative  field,  such  as 
operations  research,  economics,  engineering,  mathematics,  statistics,  or  physics 
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Military  Personnel 

Nearly  all  of  the  military  personnel  assigned  to  SMC  as  cost  analysts 
are  recent  graduates  of  the  cost  analysis  master’s  degree  program  at 
the  Air  Force  Institute  of  Technology  (AFIT).  The  minimum  curricu¬ 
lum  consists  of  four  statistics  core  classes,  six  courses  covering  subjects 
in  econometrics,  probability  and  statistics,  regression  and  time  series 
forecasting,  advanced  cost  estimating,  cost-risk  estimating,  life-cycle¬ 
costing,  economics,  engineering  economic  analysis  and  activity-based 
costing,  along  with  a  capstone  course  and  12  hours  of  thesis  research. 
In  addition,  full-time  Air  Force  students  must  complete  additional 
courses  in  simulation,  acquisition  management,  quantitative  decision¬ 
making,  and  decision  analysis.2 

Based  on  our  interviews  with  the  program  control  chiefs  and  cost 
leaders  in  the  SMC  program  offices,  these  graduates  come  to  SMC 
with  an  excellent  knowledge  of  quantitative  analysis  and  cost  analy¬ 
sis  techniques.  Their  practical  experience  in  applying  these  techniques 
and  learning  the  ins  and  outs  of  the  acquisition  process  is  typically 
acquired  on  the  job.  In  nearly  every  case,  they  were  viewed  as  being 
valued  members  of  the  cost  organizations.  However,  some  frustration 
was  also  expressed  because  by  the  time  they  had  become  highly  pro¬ 
ductive  members  of  the  team,  these  officers  frequently  rotated  out  to 
a  position  that  did  not  involve  cost  analysis,  due  to  either  short-term 
deployment  requirements  or  to  normal  military  rotation.  Anecdotal 
evidence  indicated  that,  for  most  AFIT-trained  cost  analysts,  the  future 
would  not  include  another  cost  analysis  assignment  because  of  limited 
opportunities  for  increasingly  responsible  cost  analysis  positions.  Thus, 
their  considerable  potential  in  a  hard-to-hll  career  field  was  lost  to  the 
Air  Force  after  one  (or  at  most  two)  assignments. 

Contractor  Support 

A  considerable  portion  of  the  cost  analysis  at  SMC  is  provided  by  SETA 
(also  known  as  Advisory  and  Assistance  Services)  contractors.  These 
contractors  generally  work  under  multiyear,  competitively  awarded 
contracts  that  provide  cost  analysis  and  other  FM  support  under  indi- 


2  Air  Force  Institute  of  Technology  Web  page,  2007. 
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vidual  delivery  orders.  These  delivery  orders  specify  the  nature  of  the 
support  to  be  provided  and  the  labor  hours  to  be  expended.  The  con¬ 
tractor  provides  the  required  number  of  hours,  by  labor  category,  as 
specified  in  the  delivery  order.  The  contractor  is  also  responsible  for 
planning,  managing,  and  executing  each  delivery  order,  subject  to  gov¬ 
ernment  review.  Each  delivery  order  is  funded  by  the  program  or  staff 
organization  supported. 

Based  on  our  interviews  and  head  count  data,  much  of  the  day- 
to-day  cost  analysis  work  at  SMC  appeared  to  be  done  by  support  con¬ 
tractors.  In  several  cases,  the  contractor  employees  had  considerably 
more  cost  experience  in  a  particular  program  than  did  their  govern¬ 
ment  counterparts.3  While  there  seemed  to  be  a  high  level  of  satis¬ 
faction  with  the  quality  of  contractor  support  at  SMC,  this  situation 
highlights  the  need  for  qualified  government  leadership  to  ensure  that 
the  contractor’s  priorities  and  products  are  in  fact  meeting  the  needs  of 
the  government. 

Cost  Analysis  Staffing  at  Other  Organizations 

As  part  of  our  research  into  various  approaches  to  structuring,  staffing, 
and  managing  cost  analysis  organizations,  we  interviewed  a  variety  of 
other  government  and  support  contractor  organizations.  These  organi¬ 
zations  had  approximately  similar  missions  or  environments  to  those  of 
SMC  and  thus  could  provide  experiences  and  insights  that  might  be 
relevant  to  SMC. 

To  determine  how  comparable  cost  analysis  groups  recruit,  hire, 
and  retain  their  analysts,  we  asked  the  following  organizations  about 
their  staffing  and  personnel  development  practices: 

•  The  Aerospace  Corporation 

•  Air  Force  Aeronautical  Systems  Center 

•  Air  Force  Cost  Analysis  Agency 

•  Air  Force  Electronic  Systems  Center 


3  Since  contractors  are  directly  funded  by  the  SPOs,  they  should  not  serve  as  a  main  point 
of  contact  for  cost-estimating  functions.  Doing  so  might  give  the  impression  of  bias,  whereas 
the  government  employee  not  in  the  direct  reporting  chain  may  be  more  independent. 
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•  NASA 

•  National  Reconnaissance  Office 

•  Naval  Air  Systems  Command 

•  Naval  Sea  Systems  Command 

•  Office  of  the  Secretary  of  Defense  (PA&E) 

•  Tecolote  Research,  Inc. 

In  nearly  every  case,  these  organizations  try  to  recruit  candi¬ 
dates  who  have  mathematics,  science,  or  engineering  backgrounds. 
Equally  important  was  the  breadth  of  experience  in  various  acquisi¬ 
tion  programs.  However,  all  acknowledged  that  candidates  with  these 
credentials  tend  to  be  in  short  supply.  To  address  this  shortage,  most 
organizations  used  a  combination  of  recruiting  entry-level  science  and 
engineering  graduates  and  qualified  civil  service  employees  in  other 
specialties;  career-development  through  training  and  professional  cer¬ 
tification;  rotational  assignments;  and  grade  structure.  Most  also  aug¬ 
mented  staff  capabilities  with  varying  degrees  of  contractor  support. 

NAVAIR  and  Tecolote  Research  have  formal  training  programs 
in  cost  analysis  and  the  acquisition  process  for  entry-level  cost  analysts. 
Several  other  organizations  had  some  type  of  cost  analysis  orientation 
as  part  of  their  entry-level  financial  management  training,  but  no  spe¬ 
cialized  training  focused  on  cost  practitioners.  With  these  exceptions, 
most  in-house  cost  analysis  training  is  done  “on  the  job”  by  work¬ 
ing  under  the  supervision  of  one  or  more  experienced  cost  analysts. 
Rotational  assignments  for  entry-level  analysts  and  periodic  rotation 
of  journeyman  analysts  into  positions  with  different  responsibilities 
broaden  skills  and  perspectives.  Figure  5.2  compares  the  SMC/FMC 
cost  function  to  other  comparable  organizations.  Each  bar  in  the  figure 
represents  the  total  number  of  cost  analysts  in  each  organization.  The 
blue  portion  indicates  the  number  of  analysts  located  in  a  central  cost 
organization;  the  orange  part  shows  the  number  of  analysts  reporting 
directly  to  program  directors  or  managers. 
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Figure  5.2 

Comparisons  of  SMC/FMC  Personnel  with  Other  Cost  Organizations 
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Retaining  personnel  with  skills  and  experience  that  are  in  high 
demand  is  a  challenge  for  many  government  organizations.  Most  cost 
organizations  we  interviewed  emphasized  the  importance  of  providing 
an  attractive  career  path  for  their  analysts.  Appropriate  grade  levels, 
including  nonsupervisory  GS-l4/15s,  were  considered  important  to 
remain  competitive  with  other  employment  opportunities  (including 
other  specialties  within  the  FM  career  held.)  For  example,  about  10  per¬ 
cent  of  NAVSEA,  NAVAIR4  and  NRO  cost  analysts  are  GS-15s,  and 
each  organization  is  led  by  a  member  of  the  Senior  Executive  Service 
(SES).  Several  also  attempted  to  emphasize  and  promote  the  attractive 
aspects  of  a  career  in  cost  analysis  by  such  actions  as  the  following: 

•  Exposure  to  and  participation  in  many  different  aspects  of  pro¬ 
gram  definition  and  management 


4  NAVAIR  also  supports  the  program  office  budgeting  activities,  which  are  traditionally 
accomplished  by  the  program  offices  in  many  other  organizations. 
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•  Opportunities  for  innovation,  since  few  “textbook”  approaches 
are  appropriate  for  all  estimating  situations 

•  Satisfaction  and  recognition  that  comes  from  providing  advice  to 
decisionmakers  that  is  valued  and  acted  upon 

•  The  intellectual  challenges  of  working  with  state-of-the-art  tech¬ 
nologies  and  cutting-edge  applications. 

Another  common  problem  faced  by  the  cost  analysis  organiza¬ 
tions  we  interviewed  is  obtaining  adequate  support  from  independent 
technical  experts.  Often,  the  people  most  qualified  in  a  subject  area 
are  those  already  working  for  the  program  as  government,  contrac¬ 
tor,  or  FFRDC  employees.  It  can  be  difficult  for  these  people,  who 
have  invested  their  best  efforts  and  credibility  in  developing  the  cur¬ 
rent  program  approaches,  to  distance  themselves  from  their  work  and 
objectively  assess  the  true  technical  and  programmatic  risk.  Because 
of  the  general  shortage  of  engineers  and  scientists  experienced  in  rel¬ 
evant  defense  technologies,  cost  organizations  are  using  a  variety  of 
approaches  to  obtain  this  expertise.  The  NRO  Cost  Group,  in  addition 
to  using  its  technically  oriented  government  and  contractor  cost  per¬ 
sonnel,  has  support  contracts  in  place  to  provide  independent  subject- 
matter  experts.  The  NAVAIR  Cost  Division  has  senior  cost  analysts, 
who  are  also  subject  matter  experts,  on  staff  to  participate  in  Non- 
Advocate  Reviews,  oversee  cost  research  projects,  and  provide  consult¬ 
ing  help  to  the  programs  as  needed.  (Several  of  our  interviewees  men¬ 
tioned  instances  in  which  technical  or  schedule  problems  first  raised  by 
the  cost  analysts  were  later  confirmed  and  addressed  by  the  program 
management  or  the  system  contractor.) 

Summary 

SMC  recently  implemented  a  new  organizational  structure  using  the 
wing/group/squadron  paradigm  of  the  operating  forces.  This  organiza¬ 
tion  is  intended  to  increase  the  authority  and  accountability  of  the  pro¬ 
gram  manager.  Even  before  this  latest  change,  the  cost  analysis  func¬ 
tion  has  been  migrating  from  the  center  staff  to  the  program  offices 
over  the  past  10-15  years.  These  trends  may  have  had  the  unintended 
consequence  of  weakening  the  SMC’s  cost  analysis  capability,  particu- 
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larly  in  its  role  of  providing  independent  cost  and  risk  assessments  to 
leadership  both  within  and  outside  of  the  program. 

Today,  a  mix  of  civil  service,  military,  and  support  contractor  per¬ 
sonnel  perform  SMC  cost  analysis  functions.  Within  the  Air  Force, 
cost  analysis  has  traditionally  been  considered  a  subset  of  financial 
management  for  purposes  of  organizational  structure  and  career  devel¬ 
opment.  While  there  are  areas  of  commonality,  many  functions  and 
skills  required  in  cost  analysis  differ  from  those  required  for  the  spe¬ 
cialties  of  budget  and  accounting  and  finance.  The  lack  of  senior  cost 
analysis  positions,  compared  with  those  in  other  FM  specialties,  tends 
to  discourage  civil  service  and  military  personnel  from  pursuing  the 
multiple  cost  assignments  needed  to  develop  the  in-depth  expertise 
required. 

To  gain  perspective  on  organizational,  process  and  staffing  issues, 
we  interviewed  leaders  in  other  cost  analysis  organizations  with  respon¬ 
sibilities,  functions,  and  environments  similar  to  those  at  SMC.  Nearly 
all  these  organizations  told  us  they  placed  a  high  priority  on  developing 
a  workforce  with  analytical  and  technical  skills  and  were  reasonably 
successful  in  providing  this  workforce  with  an  attractive  career  path  to 
encourage  further  development  and  retention.  They  also  differed  from 
SMC  in  that  their  cost  analysts,  while  dedicated  to  working  on  specific 
programs,  reported  to  a  strong  functional  organization  rather  than  to 
the  program  manager  they  were  supporting.  This  was  seen  as  promot¬ 
ing  objectivity,  higher  and  more  consistent  standards,  and  improved 
career  development.  Several  of  the  organizations  also  noted  the  value 
of  having  access  to  technical  experts  who  were  likewise  independent  of 
the  program  office  to  assist  in  surfacing  potential  problems  early  and 
developing  objective  assessments  of  risk. 


SMC  Cost  Analysis  Workforce  Assessment 

The  first  part  of  this  chapter  described  the  SMC  organization  and  how 
its  cost  function  compares  to  some  other  cost  functions  in  the  govern¬ 
ment.  In  this  section,  we  first  describe  the  current  cost  analysis  work¬ 
force  available  at  SMC,  grouped  by  personnel  source,  education,  expe- 
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rience,  and  unit.  We  then  detail  the  work  the  units  must  undertake  to 
complete  their  missions.  Finally,  we  model  how  the  current  staffing 
could  better  meet  projected  future  demand,  under  current  organiza¬ 
tional  principles  and  the  alternative  suggested  in  Chapter  Six. 

Data  Source  and  Survey  Instrument  Details 

Data  for  cost  analysis  workforce  supply  and  demand  were  acquired  from 
each  of  the  Acquisition  Category  I  (ACAT  I)  SPOs  at  SMC,  SMC/ 
FMC,  and  the  Space  Radar  SPO.  Two  survey  instruments  were  sent:5 
the  first  instrument  asked  cost  analysis  functional  leaders  to  identify 
all  military,  civil  service,  and  contractor  cost  and  EV  analysts;  to  detail 
their  acquisition,  cost,  and  EV  experience;  and  to  summarize  their  level 
of  education,  Acquisition  Professional  Development  Program  (APDP) 
qualification,  and  pay  grade.  The  second  instrument  asked  unit  lead¬ 
ers  to  assess  their  entire  workload,  broken  down  into  nearly  70  tasks; 
the  responses  assessed  how  frequently  each  task  comes  up,  how  long 
each  task  takes  to  complete,  and  how  many  analysts  are  required.  Note 
that  the  workforce  supply  data  do  not  contain  a  measure  of  individual 
effectiveness  or  quality.  However,  although  the  data  do  not  specify  the 
quality  of  work  performed  with  varying  numbers  or  skill  levels  of  per¬ 
sonnel,  it  is  clear  that  individuals  with  similar  workforce  characteris¬ 
tics  can  excel  at  different  tasks  and  that  many  different  combinations 
of  workers  can  form  equally  effective  teams  to  complete  acquisition 
tasks. 

RAND  received  multiple  responses  from  SMC  for  both  datasets: 
cost  analysis  workforce  supply  in  June  2006,  supply  and  demand  in 
November  2006,  and  updates  to  both  from  then  until  February  2007. 
The  earliest  supply  datasets  contained  complete  experience,  education, 
training,  and  pay  grade  information  for  all  analysts.  However,  the 
most  recent  version  accurately  reflects  total  numbers  of  personnel  only; 


5  Each  SPO  determined  the  best  means  to  obtain  these  data.  This  led  to  some  inconsistency 
in  the  interpretation  of  years  of  experience  measures,  with  some  SPOs  reporting  “acquisi¬ 
tion”  experience  to  be  equal  to  the  sum  of  cost  and  earned  value  experience,  while  other  SPOs 
treated  “acquisition”  experience  as  years  in  the  acquisition  community.  But  this  method  of 
acquiring  data  has  its  benefits,  as  well:  Each  SPO  was  able  to  provide  workforce  demand  data 
to  best  fit  its  own  scheduling  and  management  methods. 
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these  are  presented  in  Figure  5.3.  Figures  5.4  and  5.5  come  from  the 
most  recent  iteration  that  contains  a  complete  description  of  analyst 
characteristics.  Hence,  bars  for  the  latter  figures  do  not  sum  to  current 
personnel  totals.  However,  the  patterns  are  similar  across  all  iterations 
of  data. 

Cost  Analysis  Workforce  Supply 

The  current  cost  analysis  staffing  profile  of  the  Space  and  Missile  Sys¬ 
tems  Center  is  a  mix  of  civil  service,  military,  SETA  contractor,  and 
FFRDC  personnel.6 7  To  meet  expected  staffing  needs,  these  116  person¬ 
nel  are  employed  unevenly  among  FMC  and  the  SPOs.  In  Figure  5.3, 
we  see  that  in  every  unit,  nonorganic  personnel  are  the  dominant  cat- 


Figure  5.3 

Personnel  Assigned  to  Major  SMC  SPOs7 
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6  The  FFRDC  is  represented  by  The  Aerospace  Corporation  analysts  providing  technical 
assistance. 

7  Data  on  11  SETA  and  FFRDC  personnel  for  FMC  were  not  collected  in  the  survey 
instrument,  but  are  listed  in  Figure  5.3;  SETA  personnel  are  utilized  at  FMC,  but  are  gen¬ 
erally  brought  in  on  an  as-needed  basis.  Information  about  them  was  obtained  through 
interviews  with  FM  personnel.  The  rest  of  the  figures  draw  on  data  for  the  remaining  107 
personnel. 
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Figure  5.4 

Number  of  Civil  Service  Cost  Analysts,  by  Unit  and  Cost  Experience 


RAND  MG690-5.4 


egory  in  terms  of  numbers;  only  Space  Radar  has  more  military  than 
civil  service  personnel.8  As  can  be  seen  in  Figure  5.4,  which  shows  the 
number  of  civil  service  analysts  by  unit  and  level  of  cost  experience, 
some  units  have  a  mix  of  more- experienced  and  less-experienced  civil 
service  personnel  working  with  the  SETA  contractors,  whereas  Space 
Radar  has  only  one  experienced  civil  service  analyst. 

It  would  be  natural  to  show  data  on  education  level,  pay  grade, 
and  various  experience  levels  of  personnel  within  units  next,  but  in  this 
case  neither  the  education  nor  the  experience  level  of  civil  service  and- 
contractor  personnel  provides  additional  information,  since  they  are 
highly  correlated  and  have  similar  patterns.9 

However,  military  cost  analysts  currently  fit  a  different  and  quite 
narrow  pattern:  a  master’s  degree  in  cost  analysis  or  related  discipline 


8  Here  organic  refers  to  government  employees,  that  is,  both  members  of  the  civil  service 
and  military.  Nonorganic  encompasses  SETA  and  FFRDC  analysts. 

9  In  the  dataset,  EV  experience,  cost  experience,  and  acquisition  experience  are  highly  cor¬ 
related.  RAND  was  tasked  with  looking  primarily  at  cost  capabilities,  so  cost  experience  was 
chosen  as  the  primary  measure  in  this  study. 
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Figure  5.5 

Years  of  Cost  Experience  and  Source 
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with  fewer  than  five  years  of  cost  analysis  experience.  The  experience 
component  of  SMC  cost  analysts  is  presented  in  Figure  5.5,  which 
shows  the  number  of  years  of  cost  experience  by  year.  Remarkably, 
over  20  percent  of  all  personnel  have  less  than  one  year  of  experience, 
and  40  percent  have  fewer  than  five,  whereas  all  military  personnel 
have  fewer  than  five  years  of  experience.  As  described  earlier  in  this 
chapter,  the  current  career  path  of  military  officers  does  not  allow  them 
to  specialize  in  acquisition  cost  analysis  or  reward  them  for  doing  so. 
Indeed,  few  will  have  more  than  one  tour  in  cost  analysis  in  an  entire 
USAF  career. 

A  more  compact  way  to  look  at  the  same  experience  informa¬ 
tion  can  be  seen  in  Figure  5.6,  which  shows  the  number  of  analysts  by 
source.  While  roughly  ten  civil  service  personnel  have  more  than  eight 
years  of  experience,  the  vast  reservoir  of  experience — the  “institutional 
memory” — is  held  by  more  than  30  contractor  personnel  with  more 
than  eight  years  of  experience.  And,  of  the  half  dozen  contractor  per- 
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Figure  5.6 

Number  of  Analysts,  by  Source  and  Years  of  Cost  Experience 
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sonnel  with  fewer  than  three  years  of  experience,  three  have  master’s 
level  degrees  and  two  have  more  than  15  years  of  EV  experience  (details 
not  shown). 

However,  that  does  not  imply  that  the  civil  service  personnel  with 
fewer  than  three  years  of  cost  experience  do  not  also  provide  significant 
institutional  memory.  In  fact,  as  can  be  seen  in  Figure  5.7,  four  civil 
service  personnel  lack  bachelor’s  degrees;  not  shown  is  the  fact  that  one 
of  them  has  four,  and  the  other  zero  years  of  cost  experience,  yet  they 
have  17  and  20  total  years  of  acquisition  experience,  respectively. 

Figure  5.7  also  shows  that  the  vast  majority  of  military  person¬ 
nel  have  master’s  degrees,  including  degrees  in  cost  analysis  (from  the 
Air  Force  Institute  of  Technology)  or  MBA  degrees.  While  slightly  less 
than  half  of  civil  service  personnel  have  master’s  degrees,  slightly  more 
than  half  of  contractor  personnel  have  master’s  degrees.  However,  while 
half  the  civil  service  personnel  with  master’s  degrees  have  more  than 
eight  years  of  cost  experience,  three-quarters  of  contractor  personnel 
with  master’s  degrees  have  more  than  eight  years  of  cost  experience. 
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Figure  5.7 

Number  of  Cost  Analysts,  by  Source  and  Education 


RAND  MG690-S.7 


Cost  experience  and  education,  while  considered  qualitatively 
indicative  of  capability  and  performance,  do  not  provide  hard  infor¬ 
mation  about  the  ability  of  a  worker  to  perform  some  or  all  the  tasks 
required  of  an  analyst  in  the  acquisition  community.  That  is  where 
APDP10  level  might  have  been  useful.  But  APDP  is  essentially  a  man¬ 
agement  program,  not  an  independent  test  of  ability;  as  such,  it  sets 
minimum  criteria  for  education,  training,  and  experience.  As  a  result, 
APDP  levels  are  highly  correlated  with  both  degree  level  and  years  of 
cost  analyst  experience  and  cannot  serve  as  an  independent  measure  of 
effectiveness  or  quality.  Additionally,  once  at  the  GST 3  level,  almost 
all  civil  service  analysts  achieve  and  maintain  APDP  level  III,  as  can 


10  In  response  to  the  1989  Defense  Management  Review,  the  Assistant  Secretary  of  the  Air 
Force  (Acquisition)  established  an  Acquisition  Career  Development  Program  that  applies 
to  officers,  enlisted,  and  civilian  personnel  occupying  acquisition  positions.  U.S.  Depart¬ 
ment  of  Defense,  Acquisition  Career  Development  Program,  DoD  5000. 52M,  November 
1995,  established  the  specific  standards  to  be  met  by  individuals  filling  acquisition  positions. 
To  ensure  acquisition  professionals  meet  these  standards,  the  Air  Force  enacted  a  process 
through  which  they  can  be  certified  at  level  I  (basic),  level  II  (intermediate),  and  level  III 
(advanced). 
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be  seen  in  Figure  5.8.  Few  military  cost  analysts  obtain  the  experience 
necessary  to  qualify  for  APDP  level  III. 

Cost  Analysis  Workforce  Demand 

Listing  the  number  and  qualifications  of  personnel  assigned  to  units  is 
a  far  easier  task  than  describing  the  diverse  tasks  that  every  analyst  will 
perform  in  any  single  year.  Figure  5.9  sums  the  cost  analysis  workforce 
demand  provided  by  SMC,  with  the  number  of  work-hours  in  indi¬ 
vidual  task  elements  aggregated  into  categories  representing  the  broad 
mission  that  is  being  supported. 

The  cost  analysis  workload  demand  survey  sent  to  all  SPOs  was  an 
Excel  spreadsheet  listing  dozens  of  typical  tasks  a  SPO  might  undertake. 
For  each  task,  the  SPO  leader  was  requested  to  list  the  number  of  people 
assigned  to  a  task,  the  frequency  with  which  they  are  assigned  to  per¬ 
form  it,  and  the  length  of  time  it  takes  for  them  to  complete  it.  Although 

Figure  5.8 

Number  of  Organic  Cost  Analysts,  by  Pay  Grade  and  APDP  Level 


NOTE:  0-2,  0-3,  0-4,  and  0-5  correspond  to  First  Lieutenant,  Captain,  Major,  and 
Lieutenant  Colonel,  respectively. 
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the  survey  was  meant  to  be  comprehensive,  several  respondents  added 
tasks  they  believed  were  not  clearly  subsumed  under  other  categories. 

RAND  aggregated  the  individual  tasks  into  several  functions. 
For  example,  a  few  of  the  tasks  included  in  the  “estimating”  func¬ 
tion  are  “perform  annual  program  office  estimate  reviews,”  “perform 
cost-benefit  analyses,”  and  “maintain  cost  analysis  requirements 
description.” 

As  can  be  seen  in  Figure  5.9,  which  shows  data  reported  by  SMC 
and  narrowed  into  functions  (not  modeling  output),  the  vast  major¬ 
ity  of  tasks  are  related  to  major  reviews.  Estimating  is  important  for 
some  units,  and  scheduling  is  important  for  others.  Tasks  relating  to 
external  communications,  such  as  answering  the  GAO’s  inquiries, 
account  for  six  to  eight  percent  of  time  for  SBIRS,  Launch  Vehicles  and 
Ranges,  and  Space  Radar,  but  take  up  much  less  time  for  other  units; 
other  tasks,  mostly  relating  to  training,  conferences,  and  other  human 
resource  matters,  take  up  the  remainder  of  the  time. 

Figure  5.9 

Percentage  of  Work-Hours  for  Each  Unit,  by  Task 
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Cost  Analysis  Supply  Versus  Demand 

The  two  survey  instruments  can  be  used  to  compare  current  cost  anal¬ 
ysis  workforce  versus  self-reported  workload  demand.  In  the  aggregate, 
SMC  reported  that  116  organic  and  contractor  personnel  were  available 
for  work,  and  that  200  work-years  of  effort  were  demanded  of  them.  As 
can  be  seen  in  Figure  5.10,  only  Space  Radar,  with  eight  people  on  staff 
and  five  person-years  worth  of  work  demanded  of  them  has  an  adequate 
supply  of  cost  analysts  to  meet  its  workload  demand.  The  reported  data 
from  Space  Radar  suggest  a  different  imbalance  in  which  the  demand 
is  less  than  supply.  All  others  are  unbalanced.  SBIRS  and  LR  report 
that  they  need  more  than  twice  the  number  of  current  personnel;  GPS 
and  MILSATCOM  report  that  they  need  75  percent  more. 

But  these  figures  are  not  necessarily  inconsistent,  for  several  rea¬ 
sons.  First,  the  supply  figure  is  a  snapshot  of  recent  history;  contractor 
and  military  personnel  movement  into  and  out  of  SPOs  is  a  constant 
process.  Second,  the  demand  figure  is  a  projection;  the  demand  survey 
asked  extensive,  detailed,  and  difficult  questions  about  workload,  to 
which  the  respondent  had  to  apply  best  judgment  without  reference 

Figure  5.10 

Worker  Supply  and  Workforce  Demand  at  SMC,  by  SPO 
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to  comprehensive  historical  data.  The  respondents  did  not  specify  their 
risk  levels  along  with  their  point  estimates.  Hence,  the  results  are 
likely  to  be  “conservative,”  that  is,  they  are  likely  to  reflect  beliefs  about 
the  upper  bound  of  the  workload. 

Basics  of  the  Cost  Analysis  Workforce  Model11 

Snapshots  of  current  staffing  supply  and  near-term  cost  analysis  work¬ 
force  demand  cannot  provide  an  estimate  of  how  well  future  workforce 
supply  will  meet  future  workforce  demand.  To  address  concerns  about 
a  long-run  overdemand,  RAND  created  a  dynamic  monthly  forecast¬ 
ing  model  that  estimates  the  number  of  personnel  required  to  complete 
cost  analysis,  schedule,  budget,  and  EV  tasks  at  SMC,  while  assuming 
that  current  staffing  supply  will  remain  constant.  Although  it  provides 
a  reasonable  picture  at  all  time  periods,  the  model  is  especially  targeted 
at  determining  periods  of  peak  demand. 

In  the  model,  the  primary  workforce  drivers  are  major  reviews.  To 
determine  the  requirement  for  each  of  these  reviews,  RAND  developed 
a  “base  case”  for  the  number  of  analysts  required  in  a  month  during 
which  a  review  is  under  way.  For  each  type  of  major  review  (milestone 
review  or  source  selection),  the  required  set  of  analysts  with  various 
experience  levels  (0-3,  3-8,  or  8+  years  of  experience)  was  determined 
by  the  expert  judgment  of  the  RAND  team.  A  detailed  overview  of  the 
model  and  its  assumptions  is  included  in  Appendix  D. 

We  made  two  major  modifications  to  this  base  case:  (1)  the  scale 
of  the  project  as  measured  by  cost12  and  (2)  the  amount  of  time  before 
and  after  a  review  due  date.13  To  this  we  added  secondary  and  tertiary 
workforce  drivers. 


11  During  the  analysis,  a  detailed  set  of  assumptions  and  results  of  the  workforce  model 
were  sent  to  SMC  for  verification.  Except  for  concerns  that  types  of  work  were  not  initially 
addressed,  RAND  received  no  comment  about  the  accuracy  or  the  adequacy  of  its  workforce 
model. 

12  The  base  case  is  for  programs  of  less  than  $1  billion  in  total  budget.  Programs  between  $5 
billion  and  $10  billion  have  50  percent  greater  demand.  Programs  over  $20  billion  have  80 
percent  greater  demand. 

13  The  time  pattern  is  for  seven  months  before  and  one  month  after  the  due  date.  Half  a  team 
is  needed  for  initial  startup  and  documentation  cleanup.  In  between,  the  multiplier  of  the 
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The  secondary  drivers  are  continual  demands  for  a  fixed  amount 
of  budget  preparations  and  EV  calculations  that  each  unit  must  com¬ 
plete  regularly,  regardless  of  how  large  the  demands  are  from  the  pri¬ 
mary  workforce  drivers.14  The  tertiary  drivers  are  activities  that  are 
more  schedule-flexible,  such  as  training,  estimating,  scheduling,  data¬ 
base  development  activities,  and  other  studies.  In  addition,  there  are 
absences  due  to  illness,  vacation,  and  family  time. 

SMC  supplied  RAND  with  a  list  of  estimated  dates  for  all  major 
milestone  reviews  and  source  selections  in  the  near  future.  RAND 
assumed  that  three  years  would  pass  between  major  reviews,  permit¬ 
ting  the  extension  of  the  model  past  2010. 

The  outputs  of  the  model  are  shown  in  Table  5.1.  For  each  SPO 
and  FM,  the  first  column  contains  the  median  monthly  demand  for 
cost  analysts  from  January  2006  to  December  2010. 15  The  second 
column  contains  the  current  manpower  available  to  meet  demand.  The 


Table  5.1 

Cost  Analysis  Workload  Model  Runs  Summary  Results  at  Each  SPO 
and  FM  (from  January  2006  to  December  2010) 


Organization 

Median 

Demand 

Supply 

Percent  of 
Months 
75%  x  D  >  S 

Percent  of 
Time  D  >  S 

Percent  of 
Months 
125%  x  D  >  S 

MILSATCOM 

25.5 

19 

54 

100 

100 

SBIRS 

8.4 

14 

2 

13 

28 

L&R 

46.7 

18 

100 

100 

100 

GPS 

26.3 

26 

19 

51 

93 

Space  Radar 

5.6 

8 

0 

4 

23 

FMC 

32.0 

31 

0 

64 

100 

base  case  ranges  from  1  to  1.75.  The  base  case  represents  staffing  needed  between  120  and  60 
days  before  a  review  due  date. 

14  To  better  simulate  the  budget  process,  the  monthly  work  required  was  transformed  to 
peak  at  the  end  of  the  fiscal  year.  In  addition,  budget  work  was  doubled  in  calendar  year 
2009  to  meet  the  prospective  demands  of  the  next  Quadrennial  Defense  Review. 

1 '  In  this  stochastic  model,  the  demand  in  any  given  month  can  vary;  however,  the  median 
demand  across  60  months  is  relatively  stable.  Data  for  median  demand  and  the  percentage  of 
months  that  demand  exceeds  supply  are  the  mean  values  for  ten  iterations  of  the  model. 
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three  columns  on  the  right  indicate  how  frequently  different  levels  of 
demand  for  cost  analysts  exceeds  current  supply. 

To  interpret  Table  5.1:  In  roughly  4  percent  of  all  months,  the 
demand  denoted  by  the  letter  D  for  analysts  in  Space  Radar  exceeded 
supply  denoted  by  the  letter  S.  If  we  assume  that  demand  will  be  met 
with  a  staff  of  higher  quality,  effectively  lowering  demand  for  analysts 
by  25  percent,  then  Space  Radar  demand  will  never  exceed  supply. 
However,  if  we  assume  that  demand  will  be  met  with  a  staff  of  lower 
quality,  effectively  increasing  demand  for  analysts  by  25  percent),  then 
Space  Radar  demand  will  exceed  supply  in  25  percent  of  months. 

The  125  percent  column,  representing  a  cost  analysis  workforce 
with  lower  training,  experience,  and  productivity,  indicates  that  the 
number  of  analysts  needed  to  complete  work  can  be  nearly  double  the 
current  supply.16  The  75  percent  column,  representing  a  higher-quality 
workforce,  is  roughly  equal  to  current  surveyed  manpower,  indicating 
that  reorganization,  combined  with  measures  to  increase  the  produc¬ 
tivity  of  the  SMC  workforce  can  be  a  viable  alternative  to  increasing 
the  quantity  of  personnel. 

The  75  and  125  percent  figures  are  intended  to  help  address  con¬ 
cerns  that  both  supply  and  demand  numbers  are  “soft.”  First,  civil  ser¬ 
vice,  especially  military  supply,  will  constantly  be  affected  by  turn¬ 
over,  and  many  SETA  contractors  are  brought  in  on  an  as-needed  basis. 
Second,  workforce  demand  is  a  “most  likely”  estimate,  sensitive  to  any 
of  a  number  of  very  detailed  assumptions.  Absent  from  the  RAND 
model  is  any  quantification  of  the  quality  of  the  cost  analysis  workforce 
at  SMC.  Hence,  it  is  important  not  to  focus  on  the  “gap”  between  the 
cost  analysis  workforce  supply  and  demand;  for  management  purposes, 
it  would  be  better  to  think  about  how  best  to  keep  the  variable  supply 
in  rough  alignment  with  the  constantly  shifting  demand,  rather  than 
how  to  keep  a  fixed  supply  markedly  higher  than  an  exactly  deter¬ 
mined  requirement. 

One  possible  solution  for  the  periods  when  demand  is  greater 
than  supply  is  to  hire  more  SETA  contractors.  An  alternate  solution  is 
to  reorganize  or  reassign  SMC  cost  analysts  so  that  peak  loads  in  one 


16 


The  extra  25  percent  is  applied  to  all  types  of  work. 
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unit  can  be  filled  with  personnel  currently  located  in  another  unit.  The 
summary  result  of  such  a  combination  can  be  seen  in  Appendix  D, 
which  contains  a  build  chart  that  shows  the  total  workforce  demand 
by  component  for  all  of  SMC.  (The  actual  model  runs  are  included  in 
Appendix  D.) 

Figure  5.11  represents  RAND’s  best  aggregate  projection  of  future 
SMC  cost  analysis  workforce  demand,  given  the  portfolio  of  current 
programs  and  configurations.  Modeling  suggests  that  SMC  can  better 
meet  the  demand  under  our  recommended  reorganization  because  cost 
analysts  would  be  free  to  move  from  program  to  program  and  report  to 
a  central  cost  group.  The  details  of  this  organizational  structure  will  be 
discussed  in  the  next  chapter. 

The  organizational  improvements  described  in  this  document 
will  affect  the  staff  s  efficiency  at  accomplishing  cost-estimating  tasks. 
Increased  efficiency  means  fewer  hours  or  people  required  to  complete 

Figure  5.11 

Meeting  Demands  Better  Through  Reorganization 
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tasks,  decreasing  overall  workload  demand.  Decreased  efficiency  implies 
higher  workload  demand.  To  show  the  effects  of  higher  and  lower  effi¬ 
ciency  in  completing  tasks,  Figure  5.12  utilizes  the  arbitrary  75  to  125 
percent  band  around  the  modeled  workload  demand  (the  dotted  line 
in  Figure  5.11).  In  addition,  the  current  staffing  supply  (116)  and  work¬ 
force  demand  (200)  lines  are  also  drawn  across  the  figure.  We  can  see 
that  75  percent  of  forecasted  demand  exceeds  current  surveyed  man¬ 
power  for  29  percent  of  months,  while  100  percent  and  125  percent  of 
demand  always  exceed  current  supply. 

Summary 

The  first  part  of  this  chapter  reviewed  the  history  and  current  orga¬ 
nization  of  cost  analysis  functions  at  SMC.  The  rest  of  the  chapter 
accounted  for  the  cost  analysis  workforce  supply  and  demand  cur¬ 
rently  available  at  SMC.  In  aggregate  numbers  of  analysts,  the  cur- 


Figure  5.12 

Modeled  Workforce  Demand  with  75  Percent  and  125  Percent  Bounds 
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rent  SMC  workforce  is  dominated  by  support  contractors,  who  retain 
the  bulk  of  the  institutional  memory.  However,  at  least  one  experi¬ 
enced  civil  service  analyst  heads  the  cost  group  at  each  SPO.  And 
while  there  are  fewer  civil  service  personnel  than  contractors,  they  have 
similar  experi-ence  profiles.  Unfortunately,  military  analysts  have  lim¬ 
ited  career  field  opportunities,  as  seen  in  their  truncated  experience 
profile. 

SMC  reported  that  workload  demand  was  far  higher  than  work¬ 
force  supply.  However,  a  model  of  workload  that  permits  the  work¬ 
force  to  move  freely  within  SMC,  with  assumptions  vetted  by  SPO 
leaders,  suggests  that  SMC  can  better  meet  its  workload  demands 
through  reorganization  and  taking  other  measures  to  increase  the  pro¬ 
ductivity  of  its  workforce.  Providing  such  incentives  as  training,  better 
pay,  improved  working  conditions,  and  a  competitive  benefits  pack¬ 
age  could  help  attract  highly  qualified  analysts  from  other  government 
organizations  and  industry. 


CHAPTER  SIX 


Organizational  Structure  and  Other 
Organizational  Issues 


This  monograph  has  described  a  number  of  problems  endemic  to  the 
SMC  cost-estimating  function.  As  we  have  seen,  the  acquisition  pro¬ 
cess  as  a  whole  has  resulted  in  optimistic  cost  estimates.  Institutional, 
cultural,  and  budgetary  factors — particularly  the  lack  of  independence 
of  the  cost  analysts  from  the  program  offices — reduced  the  objectiv¬ 
ity  of  the  cost-estimating  process.  Less-than-optimal  organizational 
structure  and  responsibilities  led  to  problematic  estimates,  as  did  inad¬ 
equate  numbers  of  experienced  and  qualified  analysts  and  a  lack  of  rel¬ 
evant  data  and  methods  to  deal  with  the  complexities  of  space  systems. 
Limited  and  insufficient  cost,  programmatic,  technical,  and  schedule 
data,  along  with  insufficient  coordination  among  cost  analysts,  created 
problems  that  were  further  exacerbated  by  the  a  lack  of  adequate  risk- 
assessment  processes  and  methods,  including  independent  assessments 
of  programmatic,  technical,  and  schedule  issues. 

In  some  cases,  these  problems  are  caused  directly  by  certain 
aspects  of  the  SMC  organizational  structure.  In  others,  the  problems 
are  driven  by  other  factors,  but  issues  of  organizational  structure  and 
process  play  a  part.  We  took  a  two-pronged  approach  to  examining  the 
issues  and  the  alternatives.  We  first  reviewed  the  literature  to  analyze 
organizational  alternatives  and  found  that  a  key  aspect  was  whether 
staff  functions  were  organized  into  separate  departments,  whether  they 
were  integrated  into  line  organizations,  or  whether  a  hybrid  structure 
was  used.  This  is  termed  departmentalization ,  but  we  also  refer  to  it 
as  how  centralized  the  cost  organization  is.  We  then  looked  at  other 
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cost  organizations  to  see  how  they  were  organized,  and  departmental¬ 
ization  immediately  stood  out  as  a  key  aspect  of  their  organizations. 
Other  cost  organizations  also  provided  lessons  learned  that  in  many 
cases  offer  useful  insights  for  SMC.1 

This  chapter  describes  the  existing  SMC  cost  organization  in  this 
context,  reviews  the  alternatives  and  their  advantages  and  disadvan¬ 
tages,  and  finally  makes  a  recommendation  as  to  how  the  SMC  cost 
function  could  be  organized  more  effectively.  It  also  discusses  some 
issues  regarding  how  the  cost  function  should  interact  with  other  orga¬ 
nizations  inside  and  outside  of  SMC,  and  some  ideas  for  workforce 
management.  It  closes  with  a  description  of  the  aspects  of  a  successful 
organizational  change. 


Issues  of  Cost  Analysis  Organizational  Design 

The  problems  we  identified  that  interfered  with  the  production  of  accu¬ 
rate  cost  estimates  are  to  a  great  extent  related  to  issues  of  organiza¬ 
tion — structure  and  processes.  This  section  briefly  describes  some  of 
these  issues  to  provide  context  for  a  better  understanding  of  the  chal¬ 
lenges  SMC  faces.  Those  challenges  are  not  unique  to  SMC;  how  to 
organize  work  has  been  the  subject  of  investigation  for  hundreds  of 
years.  Adam  Smith’s  1776  discussion  of  pin  manufacturing  is  an  early 
example.2  Richard  Scott  says,  “One  of  the  most  difficult  and  critical 
of  all  decisions  facing  organizations  is  how  work  is  to  be  divided — 
what  tasks  are  assigned  to  what  roles,  roles  to  work  units,  and  units  to 
departments.”3  The  question  of  how  to  organize  is  key. 

An  important  insight  from  the  literature  is  the  concept  of  depart¬ 
mentalization  as  a  means  to  organize  work.  Should  support  functions 
be  in  separate  departments?  Or  should  they  be  part  of  the  product  (pro- 


1  We  focused  on  learning  lessons  on  what  to  do  and  what  not  to  do  from  working-level  cost 
analysis  groups — in  particular,  ASC,  ESC,  NAVAIR,  NAVSEA,  and  NRO. 

2  Adam  Smith,  An  Inquiry  into  the  Nature  and  Causes  of  the  Wealth  of  Nations,  1776. 

3  W.  Richard  Scott,  Organizations:  Rational,  Natural  and  Open  Systems,  5th  ed.,  Upper 
Saddle  River,  N.J.:  Prentice  Hall,  2003. 
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gram)  groupings  within  the  organization?  We  will  see  that  cost  analysis 
is  a  necessary  function  in  every  program  office,  yet  it  benefits  from  a 
high  level  of  central  coordination  to  ensure  that  all  analysts  have  access 
to  the  latest  tools  and  the  most  up-to-date  data — resources  that  should 
be  shared  among  all  SMC  analysts.  Also,  cost  analysts  are  in  a  distinct 
career  held  with  highly  specialized  knowledge  that  comes  from  years  of 
experience  best  learned  from  other  experts.  The  costs  of  coordination 
across  units  without  centralized  organization  are  relatively  high.  For 
analysts  to  stay  as  current  as  possible,  these  costs  would  need  to  be  born 
by  every  distributed  product  organization.  This  would  call  for  some 
degree  of  departmentalization  to  keep  cost  analysts  coordinated. 

More  formally,  the  organizational  literature  offers  many  different 
approaches  for  structuring  work.  Daft  lays  out  four  basic  alternatives.4 
A  functional  structure  divides  distinct  functions  into  individual  depart¬ 
ments,  which  are  coordinated  through  managers  above  the  depart¬ 
ments  in  the  corporate  hierarchy.  (Coordination  of  efforts  is  a  major 
issue  in  any  organizational  design.)  A  divisional  grouping  divides  the 
organization  into  product  lines,  each  of  which  has  representatives  of  all 
functions  (e.g.,  manufacturing,  marketing,  finance)  located  within  it 
and  reporting  to  its  management.  A  geographic  grouping  is  similar,  but 
divisions  are  made  by  area  rather  than  by  product,  so  that  each  division 
might  have  multiple  products.  A  multifocused  grouping,  which  involves 
a  structural  form  usually  called  a  matrix  or  hybrid  structure,  cuts  across 
two  of  the  above  groupings. 

How  do  these  alternatives  map  to  the  cost  function  at  SMC?  Our 
interviews  revealed  that  the  divisional  alternative  most  closely  char¬ 
acterizes  current  operations.  Numerous  departments  (SPOs)  focus 
on  specific  products,  with  integrated  cost  and  other  functions.  Daft 
cites  research  indicating  that  the  weaknesses  of  this  approach  include 
the  lack  of  economies  of  scale  for  functions  (cost-estimating  would 
be  included  here)  and  reductions  in  deep  functional  competence  and 


4  Richard  L.  Daft,  Organizational  Theory  and  Design,  6th  ed.,  1998,  Cincinnati,  Ohio: 
Southwestern  College  Publishing,  1998.  This  textbook  provides  a  useful  summary  of  research 
conducted  by  many  original  sources.  However,  the  topic  is  an  exceedingly  rich  one  that  is 
difficult  to  describe  completely  in  one  source.  Scott’s  (2003)  description  of  the  field  is  another 
useful  source  for  understanding  these  issues. 
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technical  skills.5  Coordination,  integration,  and  standardization  across 
departments  are  also  difficult.  These  are,  in  fact  problems  that  SMC 
has  faced.  In  our  interviews,  we  ascertained  where  most  significant  cost 
analysis  work  responsibilities  are  currently  performed  at  SMC.  These 
are  displayed  in  Table  6.1. 

As  can  be  seen  from  Table  6.1,  SMC  is  heavily  program-centric, 
with  most  of  the  cost  analysis  functions  performed  in  the  SPOs  and 
some  participation  by  the  FMC  staff.  The  two  items  with  question 
marks  were  those  where  it  was  not  clear  who  had  overall  responsibility 
for  accomplishing  the  task.  In  some  cases,  the  SPO  was  doing  them;  in 
others,  it  was  the  FMC  staff. 


Table  6.1 

Where  Cost  Analysis  Responsibilities  and  Tasks  Are 
Currently  Found 


Work  Responsibilities  and  Products 

SMC/FMC 

SPOs 

Program  acquisition  strategy,  technical 
baseline  (CARD) 

X 

Milestone  estimates,  cost  proposal 
evaluations 

X 

Budget  estimates,  excursions,  SAR  input 

X 

EV  analysis  +  IBRs 

X 

Develop  cost  models,  tools,  and  cost, 
technical  and  program  databases 

? 

? 

Represent  cost  function  to  outside 
organizations 

X 

Recruitment,  training,  career 
development  of  cost  analysts 

? 

? 

NOTES:  CARD  =  Cost  Analysis  Requirements  Description.  An 
x  means  that  the  task  takes  place  within  the  relevant 
organization.  A  question  mark  (?)  means  that  RAND  was  not 
able  to  ascertain  where  the  work  took  place. 


5  Daft,  1998,  p.  219. 
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Structure  of  Other  Cost  Organizations 

We  interviewed  analysts  and  managers  associated  with  other  military 
cost  organizations  both  inside  and  outside  the  Air  Force.6  The  nature 
and  availability  of  data  prevented  us  from  linking  organizational  struc¬ 
ture  to  the  success  of  cost-estimating  outcomes.  (That  is,  does  one  type 
of  organizational  structure  produce  more  accurate  estimates?  Program 
differences  may  make  this  comparison  less  meaningful  as  well,  since 
comparing  vastly  different  classes  of  programs  may  not  provide  any 
real  insights  into  the  challenges  of  estimation.) 

Figure  6.1  compares  the  number  of  SMC  cost  analysts  with  those 
in  other  Air  Force  organizations,  such  as  ASC  and  ESC,  as  well  as  with 
those  in  non-Air  Force  organizations,  such  as  NRO,  NAVSEA,  and 
NAVAIR.  The  blue  portion  of  each  bar  shows  the  number  of  analysts 
reporting  to  a  centralized  cost  organization,  and  the  orange  portion 

Figure  6.1 

Comparisons  of  SMC/FMC  Personnel  with  Other  Cost  Organizations 
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systems. 
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represents  the  number  of  analysts  reporting  directly  to  program  direc¬ 
tors  (PDs).  As  the  figure  shows,  the  number  of  analysts  reporting  to 
the  PD  is  much  higher  for  SMC  and  ESC.  Further,  the  figure  also 
indicates  that  the  number  of  analysts  reporting  to  a  cost  organization 
is  smaller  at  SMC  than  in  the  non— Air  Force  cost  organizations 

In  fact,  one  can  claim  that  the  SMC  cost  capability  is  currently 
situated  primarily  within  the  SPOs.  There  is  a  very  small  centralized 
FMC  staff  that  plays  a  minimal  role  in  creating  cost  estimates.  It  has 
a  GS-15  as  its  leader  and  two  GS-l4s  as  the  deputies,  one  in  charge  of 
cost  estimation  and  the  other  in  charge  of  EV  analysis.  A  proposed 
new  FMC  organization  would  be  composed  of  26  analysts,  including 
ten  civil  service  and  five  military,  located  within  this  department  and 
reporting  to  its  leadership.  The  vast  majority  of  the  actual  cost  analysis 
is  now  performed  in  the  program  offices  themselves  by  analysts  who 
report  to  the  SPO  leadership.  ESC  has  a  similar  structure  and  has  a 
GS-15  as  the  head  of  the  cost  staff. 

NRO  and  NAVAIR  have  highly  centralized  cost  departments 
that  perform  work  for  program  offices  as  needed.  Very  few  cost  ana¬ 
lysts  report  directly  to  the  PM.  Both  of  these  organizations  have  about 
10  percent  of  their  cost  staff  in  GS-15  positions. 

ASC  has  a  hybrid  structure:  Some  cost  analysts  are  located  in  the 
SPOs,  working  on  budget,  EVM,  and  other  day-to-day  cost  require¬ 
ments.  Others  are  located  in  a  small,  centralized  staff,  which  is  respon¬ 
sible  for  producing  cost  estimates  for  major  milestone  reviews.  There 
are  two  GS-15  level  analysts.  While  the  total  number  of  analysts  is 
small,  ASC  analysts  are  generally  very  senior  and  experienced. 

Interviews  conducted  with  the  other  organizations  led  to  several 
observations  that  were  shared  by  most,  if  not  all  of  them.  The  gen¬ 
eral  agreement  was  that  the  cost  function  must  be  insulated  from  the 
bureaucratic  pressures  of  the  programs  if  analysts  are  to  provide  objec¬ 
tive  estimates.  For  those  organizations  within  the  Air  Force,  there  was 
concern  that  the  “wing,  group,  squadron”  organization  will  erode  this 
insulation  and  hence  the  required  objectivity  of  the  cost  analysts.  We 
also  learned  that  support  contractors  are  useful  in  many  ways.  They 
provide  analytical  support  ranging  from  major  estimates  to  regular 
budget  work.  However,  they  should  be  managed  by  qualified  govern- 
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ment  estimators.  Usually,  program  offices  fund  the  contractor  support, 
which  could  hinder  contractors’  objectivity  because  they  may  be  under 
subtle  or  direct  pressure  to  shape  estimates  to  please  their  customer. 

Another  finding  was  that  efforts  should  focus  on  sound  initial 
estimates  to  provide  the  foundation  for  later  work.  They  are  far  pref¬ 
erable  to  efforts  spent  on  multiple  estimate  reviews  (“build  in  quality 
rather  than  inspect  in  quality”).  Generally,  reviews  take  the  initial  esti¬ 
mates  as  their  starting  points,  so  major  weaknesses  in  the  initial  esti¬ 
mate  may  not  be  corrected  during  the  estimate  review  process.  There¬ 
fore,  initial  estimates  need  emphasis  and  resources.  In  addition,  a  few 
well-qualified,  experienced  analysts  will  likely  be  much  more  effective 
than  many  inexperienced  ones. 

The  other  cost  organizations  also  indicated  that  the  mainte¬ 
nance  of  a  historical  track  record  of  estimates  is  a  key  management 
tool.  Having  a  detailed  estimate  to  reference  builds  credibility,  par¬ 
ticularly  if  the  final  program  budgeted  amount  is  significantly  lower 
than  the  original  objective  cost  estimate.  They  can  then  counter  claims 
of  bad  cost  estimation,  which  will  increase  their  credibility  as  creators 
of  reasonable  estimates.7  A  track  record  also  allows  for  the  long-term 
assessment  of  individual  analysts  and  the  accuracy  of  analytical  tools. 
A  track  record  maintained  over  time  creates  a  stronger  sense  of  owner¬ 
ship  over  the  estimate  that  might  motivate  a  more  careful  original  esti¬ 
mate.  The  other  cost  organizations  also  have  found  that  the  identifica¬ 
tion  of  major  sources  of  technical  risk  early  in  the  program  is  critical 
to  estimating  success.  Whatever  approach  is  used  to  measure  technical 
risk  should  be  robust  and  should  reflect  the  difficulties  of  developing 
individual  subsystems  as  well  as  integrating  them  into  the  final  system. 
Our  interviews  also  showed  that  many  successful  cost  analysts  have  a 


7  One  reviewer  of  this  document  raised  the  concern  that  cost  analysts  may  artificially  inflate 
their  initial  estimates  to  avoid  the  risk  of  underestimation.  However,  if  the  cost  estimates  end 
up  being  greater  than  the  final  actual  cost,  it  could  be  tracked  as  a  problematic  cost  estimate, 
just  as  an  underestimate  would  be.  A  greater  danger  is  that  program  managers  lack  incentives 
to  underrun  costs,  so  if  a  budget  was  based  on  an  overestimation  and  was  greater  than  what 
was  really  needed,  program  managers  may  spend  more  money  than  necessary  and  resources 
would  not  be  most  efficiently  allocated  for  the  overall  Air  Force.  Offering  program  managers 
incentives  to  underrun  their  cost  estimates  could  counter  this  problem. 
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technical  background.  As  will  become  clear,  these  bndings  helped  us 
develop  our  recommendations  for  SMC. 


Alternatives  for  SMC  and  Our  Recommendations 

What  do  these  structural  alternatives  mean  for  SMC?  The  three  alter¬ 
natives — a  separate  centralized  cost  department,  decentralized  ana¬ 
lysts,  and  a  hybrid  structure — all  have  different  implications  for  the 
SMC  cost  function. 

Separate  Department 

The  first  alternative  is  a  separate  cost  department  managed  as  part 
of  SMC’s  FM  organization.  In  this  structure,  all  cost  analysts  would 
report  to  the  cost  staff.  SPOs  would  have  separate  analysts  to  develop 
recurring  budgets  and  EV  estimates  and  analyses.  Figure  6.2  portrays 
a  centralized  cost  organization. 

Having  a  separate  cost  department  offers  numerous  strengths. 
Most  important,  this  structure  should  maximize  the  objectivity  and 
independence  of  the  cost  analysts.  Analysts  would  report  to — and  have 
their  performance  reviews  conducted  by — the  financial  management 
chain  of  command  instead  of  the  program  office  structure.  Being  offi¬ 
cially  part  of  the  program  offices  has  the  potential  for  both  overt  and 

Figure  6.2 
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indirect  pressures.  Presumably,  analysts  would  feel  less  overt  pressure 
from  the  program  office  to  keep  their  estimates  low  to  increase  the  like¬ 
lihood  that  programs  would  be  funded  initially. 

A  strong  cost  department  can  offer  an  effective  senior  voice  to 
represent  the  cost  community  to  SMC  leadership  and  to  external 
organizations.  The  head  of  the  department  should  be  of  sufficiently 
high  rank — SES  level — to  represent  the  cost  function  effectively  in 
discussions  with  other  senior  leaders,  such  as  general  officer  program 
managers.  Strong  leadership  in  the  department  can  obtain  necessary 
resources  for  the  cost  function,  influence  SMC  policies  that  affect  the 
cost  function,  and  so  forth.  For  example,  it  could  foster  better  inter¬ 
actions  with  the  staff  engineering  function,  so  that  cost  analysts  get 
the  required  technical  assistance.  Strong  leadership  in  the  cost  depart¬ 
ment  can  also  represent  SMC  cost  analysts  at  external  functions,  which 
may  control  resources  or  other  benefits,  such  as  the  SAF/FMC  and  the 
AFCAA  staffs.  Finally,  strong  leadership  can  interact  effectively  at  cost¬ 
estimating  conferences,  such  as  the  DoD  Cost  Analysis  Symposium 
(DoDCAS)  and  meetings  of  the  Society  of  Cost  Estimation  and  Ana¬ 
lysts  (SCEA),  with  the  FFRDCs,  and  so  forth. 

A  distinct  cost  department  also  has  career  development  implica¬ 
tions.  A  central  cost  organization  has  the  potential  to  offer  more-effec¬ 
tive  career  development  through  managed  assignments  and  training. 
It  could  provide  a  resource  for  cost  analysts  looking  to  improve  their 
skills  and  could  offer  a  clearer  hierarchy  for  promotions.  It  could  better 
manage  hiring,  training,  and  other  aspects  of  career  development.  It 
could  provide  a  structure  for  sharing  new  and  useful  tools  and  tech¬ 
niques  and  lessons  learned,  for  the  benefit  of  all. 

A  related  benefit  is  the  development  of  an  expert  ability  to  evalu¬ 
ate  analysts  on  the  quality  of  their  analysis  as  well  as  a  way  to  capture 
analyses  and  compare  estimates  with  eventual  program  performance. 
This  would  give  analysts  a  powerful  reason  to  continually  improve 
their  estimates. 

However,  a  separate  department  also  has  distinct  disadvantages. 
First,  it  may  make  cost  analysts  less  accountable  to  the  program  man¬ 
agers,  who  might  not  be  able  to  provide  incentives  for  timely  perfor¬ 
mance.  Second,  SMC  operates  in  a  larger  Air  Force  context.  The  Air 
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Force  is  going  to  a  “wing,  group,  squadron”  organizational  construct 
that  is  designed  to  give  program  managers  more  control  over  resources. 
A  separate  cost  department  directly  conflicts  with  this  structure. 

Third,  unless  cost  analysts  are  located  in  the  program  offices  and 
have  frequent  opportunities  to  interact  with  engineers  and  other  SPO 
personnel,  they  may  know  less  about  specific  programs  and  have  fewer 
technical  insights.  Space  systems  are  complex  and  are  growing  ever 
more  so,  making  technical  issues  a  significant  driver  of  cost.  Under¬ 
standing  the  complexities  of  individual  systems  is  key  to  creating  good 
estimates,  and  being  in  the  SPO  on  a  continuous  basis  for  some  period 
of  time  facilitates  that  understanding. 

Finally,  it  may  be  difficult  to  find  funding  for  a  centralized  depart¬ 
ment.  The  program  offices  would  have  to  pay  additional  “taxes”  for  cer¬ 
tain  costs  incurred  by  a  central  cost  function.  Whether  this  total  would 
be  higher  than  current  “taxes”  is  unknown,  but  we  would  expect  there 
to  be  some  negotiation. 

Full  Integration  with  Line  Organization 

In  the  second  alternative,  most  or  all  of  the  cost  analysts  would  be 
hired  by,  managed  by,  and  located  within  individual  program  offices, 
with  little  functional  integration  with  analysts  at  other  SPOs.  The 
central  FM  organization  would  have  few  analysts.  (This  most  closely 
represents  the  current  FMC  structure  at  SMC.)  Again,  this  approach 
has  both  benefits  and  limitations.  In  many  cases,  these  are  the  reverse 
of  the  benefits  and  limitations  of  the  centralized  structure.  Figure  6.3 
depicts  the  second  organizational  structure.  Integration  with  the  line 
organization  offers  PDs  direct  control  over  cost  analysis  resources.  PDs 
can  set  the  priorities  and  manage  the  work  of  the  analysts  within  their 
divisions.  One  implication  is  that  analysts  may  thus  be  more  responsive 
since  they  are  supporting  the  person  to  whom  they  directly  report. 

A  second  and  important  benefit  of  this  approach  is  that  long-term 
experience  within  a  SPO  could  enhance  the  technical  and  program¬ 
matic  knowledge  of  cost  analysts  on  that  system.  As  we  have  described, 
space  systems  are  extremely  complex  technically.  Developing  a  deep 
understanding  of  the  technical  issues  will  help  analysts  develop 
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their  cost  estimates  by  giving  them  insight  into  what  the  most  appro¬ 
priate  estimating  tools  or  analogous  systems  might  be. 

However,  full  integration  within  the  SPOs  also  poses  risks.  The 
most  significant  concern  is  the  potential  effect  on  the  objectivity  of  cost 
estimates.  This  may  be  due  to  direct  or  indirect  pressures.  Directly,  the 
PD  may  try  to  sway  the  cost  estimate  to  fit  the  expected  budget  avail¬ 
ability  or  overall  cost  goals.  One  subtle  metric  for  success  for  PDs  is 
whether  their  programs  are  funded.  If  they  are  concerned  that  cost  esti¬ 
mates  are  too  high,  they  may  fear  that  they  will  not  be  able  to  convince 
the  Air  Force  or  Congress  to  justify  spending  as  much  as  expected  to 
acquire  that  particular  capability.  As  a  result,  they  may  pressure  their 
cost  analysts  to  manage  the  estimating  process  so  as  to  generate  opti¬ 
mistic  numbers.  Although  this  may  not  be  a  directed  “estimate-to- 
budget”  order,  it  could  amount  to  the  PD  arguing  over  the  techni¬ 
cal  details  of  the  estimate  until  enough  programmatic  adjustments  are 
made  that  the  estimate  itself  falls  within  the  desired  budget  amount. 
And  if  PDs  hire  and  manage  the  careers  of  estimators  within  their  pro¬ 
grams,  they  can  reward  and  punish  compliant  or  recalcitrant  analysts. 

The  lack  of  a  strong  central  cost  department  removes  a  central 
voice  for  a  number  of  policy  issues.  These  would  include  data  collection 
and  data  sharing  with  other  organizations,  best  practices  in  estimating 
tool  development  and  tool  use,  and  other  issues  regarding  estimating 
approaches.  Without  a  clearinghouse  to  enhance  the  sharing  of  these 
issues  among  SPOs,  estimators  may  not  engage  in  the  continuing  edu¬ 
cation  necessary  to  remain  abreast  of  the  best  new  approaches.  And  the 
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lack  of  a  strong  central  voice  for  cost  may  mean  that  the  objective  cost 
perspective  is  not  adequately  represented  to  SMC’s  PEO. 

A  last  concern  is  the  limitation  on  career  development  and  pro¬ 
gression  for  individual  cost  analysts.  While  estimators  might  be  quite 
successful  within  their  SPOs,  they  may  not  have  the  same  kind  of 
career  opportunities  that  derive  from  a  more  formal  career  manage¬ 
ment  approach.  A  separate  cost  department  can  offer  a  defined  career 
track  with  broader  training  and  development  opportunities.  The  size  of 
SPOs  or  the  program’s  position  in  its  life  cycle  may  limit  the  number 
of  cost  estimators  and  limit  lateral  experience  reassignments  or  even 
advancement.  Certainly,  estimators  can  apply  for  other  jobs  within 
SMC,  but  until  those  jobs  open  up  there  may  be  little  reason  to  invest 
in  learning  new  skills  that  may  not  apply  to  their  current  programs. 
Having  a  centralized  department  that  takes  a  proactive  approach  to 
career  development  offers  analysts  a  more  distinct  career  path  by  pro¬ 
viding  a  wider  range  of  experience  and  training  opportunities. 

Hybrid  Structure 

The  third  alternative  offers  some  of  the  features  of  both  the  centralized 
cost  department  and  the  decentralized  integration  of  analysts  with  line 
organizations.  Under  this  approach,  a  new  FM  would  have  oversight 
over  all  cost  analysts,  with  many  assigned  to  the  central  FM  cost  staff.8 
Other  cost  analysts  would  be  located  in  SPOs  but  report  to  FM.  Bud¬ 
geting  and  EV  analysts  would  be  located  within  the  SPOs  and  report 
to  the  PM.  This  approach  has  many  of  the  benefits  of  both  approaches 
described  above,  with  a  few  limitations  of  its  own.  Figure  6.4  displays 
such  an  organizational  structure. 

In  this  structure,  the  cost  staff  would  be  responsible  for  pro¬ 
ducing  major  cost  estimates  for  milestone  reviews,  source  selections, 
Nunn-McCurdy  breaches,  or  other  specific  cost  estimates  requested  by 
management  at  all  levels.  The  SPO  analysts  would  be  responsible  for 
maintaining  and  updating  the  estimates  between  major  reviews,  devel¬ 
oping  budgetary  estimates  required  by  the  Planning  Programming  and 


8  We  italicize  the  word  “new”  here  to  emphasize  that  we  are  talking  about  a  new  organiza- 
tion  with  a  different  mission  and  function  from  those  of  the  current  FM  organization. 
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Budgeting  System  (PPBS),  and  periodic  EY  analyses.  This  approach 
maximizes  flexibility  in  moving  analysts  to  meet  the  peak  workloads 
of  individual  SPOs.  If  cost  analysts  are  “owned”  by  the  central  cost 
department,  then  the  department  can  deploy  and  redeploy  expert  ana¬ 
lysts  as  needed  to  meet  major  estimating  requirements. 

Under  this  approach,  the  centrally  located  analysts  would  be  less 
subject  to  direct  or  subtle  pressures  from  PDs  to  put  a  positive  spin  on 
their  estimates  and  probably  better  able  to  produce  objective  estimates. 
One  benefit  of  this  approach  would  be  that  FM  analysts  who  are  truly 
objective  could  eliminate  the  need  for  a  separate  ICE  within  SMC, 
and,  once  their  credibility  is  established,  possibly  within  the  Air  Force 
as  well. 

The  central  FM  cost  staff  could  have  oversight  over  cost  analysts 
and  effectively  manage  professional  standards  and  careers.  The  central 
organization  could  develop  a  recruitment  plan  that  features  a  regular 
pipeline  of  junior  analysts  into  the  cost  staff  as  part  of  their  initial  train¬ 
ing  and  development  as  the  expert  cost  analysts  of  the  future.  The  cost 
department  could  also  standardize  data  collection  and  best  practices  in 
estimating  tools  and  techniques  and  could  work  with  other  organiza¬ 
tions  to  share  data  and  insights  into  best  practices  in  cost-estimation 
approaches.  It  could  maintain  a  track  record  of  analyst  performance, 
thus  creating  a  further  incentive  for  objectivity  and  independence. 
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Analysts  who  are  located  at  the  SPOs  doing  budget  and  other 
frequent  tasks  would  develop  in-depth  programmatic  and  technical 
knowledge  that  they  could  share  with  analysts  from  the  central  organi¬ 
zation  who  are  on  assignment  at  the  SPO  to  work  on  major  milestone 
and  other  significant  estimating  events. 

However,  this  approach  does  not  mesh  with  the  “wing,  group, 
squadron”  structure  that  characterizes  the  Air  Force’s  preferred  ap¬ 
proach  to  designing  organizations,  an  approach  that  gives  the  maxi¬ 
mum  amount  of  control  to  line  managers,  such  as  PDs.  The  limitations 
on  their  control  of  estimates  may  make  it  difficult  to  win  PD  support. 
This  may  be  an  inherent  limitation  on  the  ability  of  SMC  to  make  a 
change  toward  a  hybrid  structure.  Furthermore,  the  issue  of  funding 
for  a  central  organization  arises  again  in  this  structure.  PDs  would  have 
to  agree  to  be  taxed  to  support  centralized  analysts  unless  a  separate 
budget  could  be  established  as  part  of  the  PPBS  process. 

Recommendation 

Nevertheless,  for  reasons  that  follow,  our  recommendation  is  that  SMC 
adopt  this  hybrid  approach.  In  our  view,  this  has  the  most  potential 
benefits  and  the  fewest  limitations.  In  particular,  increasing  the  objec¬ 
tivity  of  the  analysts  performing  major  cost  estimates  will  improve  the 
reliability  of  the  estimates  and  SMC’s  reputation  as  an  organization 
whose  cost  numbers  can  be  trusted.  This  kind  of  change  will  require 
significant  support  from  senior  SMC  leadership,  as  we  discuss  below. 

We  also  provide  recommendations  on  where  work  should  be  per¬ 
formed.  Tasks  should  be  conducted  within  the  SPO  when  the  focus  is 
on  program  execution,  where  changing  priorities  or  rapid  response  are 
common,  for  functions  required  to  manage  the  day-to-day  activities  of 
the  program,  where  the  official  position  for  effective  interaction  with 
other  SPO  personnel  is  needed,  and  where  processes  are  unique  to  the 
program. 

Tasks  should  be  performed  in  the  cost  staff  when  nonadvocate  anal¬ 
ysis  is  a  priority,  experienced  government  leadership  is  required,  econo¬ 
mies  of  scale  and  flexibility  in  assignments  are  desired,  skill  sets  and  tasks 
are  outside  the  SPO’s  primary  mission,  and  workload  and  priorities  are 
generally  predictable.  Table  6.2  revises  the  cost  analysis  responsibilities 
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Table  6.2 

Where  Cost  Analysis  Responsibilities  and  Tasks  Should  Be 
Performed  Under  a  Future  Hybrid  Structure 


Work  Responsibilities  and  Products 

SMC/FMC 

SPOs 

Program  acquisition  strategy,  technical 
baseline  (CARD) 

P 

L 

Milestone  estimates,  cost  proposal  evaluations 

L 

P 

Budget  estimates,  excursions,  SAR  input 

R 

L 

EV  analysis  +  IBRs 

P 

L 

Develop  cost  models,  tools,  and  cost,  technical 
and  program  databases 

L 

P 

Represent  cost  function  to  outside 
organizations 

L 

P 

Cost  analysts  recruitment,  training,  career 
development 

L 

P 

KEY:  L  =  leads  the  effort;  P  =  participates  in  the  effort;  R  =  reviews 
the  product  or  output  of  the  effort. 


previously  shown  in  Table  6.1  as  they  are  currently  being  accomplished 
at  SMC  by  rearranging  them  into  the  recommended  hybrid  structure. 

As  Table  6.2  shows,  the  SMC  cost  staff  is  given  the  responsi¬ 
bilities  and  tasks  that  can  benefit  most  from  a  centralized  structure, 
where  staffing  and  skill  level  leveling  is  critical,  where  policy  decisions 
or  more  standardization  are  desirable  (cost  models,  tools,  databases, 
and  documentation),  or  for  responsibilities  not  germane  to  the  primary 
functions  of  the  SPOs  (such  as  recruiting  and  training).  But  commu¬ 
nication  and  participation  by  the  SPOs  and  the  FM  staff  in  all  these 
responsibilities  would  be  desirable  as  part  of  the  hybrid  structure. 


Other  Organizational  Issues 

Workforce  Recommendations 

We  also  offer  an  outline  suggestion  for  the  management  of  the  cost 
analyst  workforce  to  support  the  new  approach.  Our  alternative  gives 
more  responsibility  to  the  central  FM  cost  organization  for  managing 
recruitment,  training,  and  career  development.  This  is  a  new  area  for 
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FM  but  one  that  can  be  done  effectively  with  the  appropriate  support. 
The  key  will  be  close  interaction  with  SMC’s  human  resources  function 
(both  military  and  civil  service  personnel  functions)  to  gain  support  for 
any  alternate  approach,  both  to  understand  the  technical  and  adminis¬ 
trative  issues  involved  in  career  management  and  to  make  sure  that  all 
the  legal  requirements  are  met.  As  we  will  describe,  these  recommen¬ 
dations  include  having  a  strong  group  of  expert  analysts  located  at  the 
FM  central  cost  department  to  be  deployed  as  needed  to  the  SPOs  to 
help  with  major  estimates;  a  group  of  mid-level  analysts  located  at  pro¬ 
gram  offices  to  provide  day-to-day  support  to  the  SPOs  while  getting 
more  in-depth  technical  and  programmatic  insights  into  specific  space 
systems;  and  a  group  of  junior  analysts  on  the  FMC  staff  being  trained 
by  senior  analysts  and  working  on  developing  their  cost  analysis  skills, 
learning  new  tools,  and  maintaining  existing  databases. 

A  “cradle-to-grave”  (or  “recruitment-to-retirement”)  approach  to 
managing  cost  analysts  might  take  the  following  path.  The  first  step 
is  recruitment.  One  of  the  difficulties  in  recruitment  is  the  fact  that 
SMC  is  located  in  Los  Angeles,  which  has  a  very  high  cost  of  living 
and,  based  on  our  interviews  at  SMC,  some  perceived  concerns  about 
the  quality  of  the  local  public  schools.9  This  may  create  difficulties  in 
recruiting  skilled  analysts  from  outside  the  LA  area,  but  there  is  a  large 
pool  of  potential  employees  who  have  attended  local  schools  or  who 
have  family  in  the  area  or  other  reasons  for  wanting  to  be  based  in  Los 
Angeles. 

We  recommend  recruiting  analysts  from  local  universities,10 
including  the  University  of  California— Los  Angeles,  the  University  of 
Southern  California,  the  University  of  California-Irvine,  and  others. 
There  is  a  wide  enough  range  of  high-quality  academic  institutions  in 
the  area  for  SMC  to  get  a  diverse  group  of  recruits,  and  SMC  could 
aim  for  getting  new  employees  from  a  range  of  institutions  and  depart- 


9  This  is  based  on  hearsay  evidence  from  interviews  conducted  at  SMC  rather  than  on  a 
systematic  assessment  of  LA  schools,  so  it  should  not  be  taken  as  an  official  evaluation  by 
RAND. 

10  We  suggest  that  SMC  focus  on  these  schools  because  of  the  potential  difficulty  of  recruit¬ 
ing  analysts  to  the  high-cost  Los  Angeles  area.  Those  who  already  live  in  Los  Angeles  may  be 
more  likely  to  view  the  benefits  as  outweighing  the  costs. 
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ments  to  get  varied  backgrounds  that  would  allow  for  a  diversity  of 
perspectives  and  skills.  Of  particular  value  would  be  students  who  have 
technical  engineering  backgrounds,  because  they  would  be  most  likely 
to  understand  the  complex  technical  issues  involved  in  space  systems 
and  would  also  have  the  quantitative  skills  needed  for  cost  analysts. 
Graduates  with  an  accounting  or  finance  background  with  technical 
aptitude  (and  perhaps  the  willingness  to  pursue  a  technical  graduate 
degree)  would  be  a  strong  second  choice,  as  would  those  with  applied 
math,  economics,  or  other  strongly  quantitative  fields  of  study. 

Yearly  numbers  of  recruits  would  vary  depending  on  forecasted 
future  needs  of  programs,  as  well  as  retirements  and  other  analysts 
leaving  the  FM  workforce.  Ideally,  recruiters  would  try  to  hire  at 
least  a  couple  of  analysts  every  year  to  maintain  a  pipeline  of  junior 
personnel  to  keep  up  ties  with  the  local  university  career  placement 
departments. 

Junior  analysts  would  be  brought  in  to  work  in  the  central  cost 
department  with  the  senior  analysts  located  there.  They  could  be 
deployed  to  the  SPOs  side  by  side  with  senior  analysts  working  on 
major  estimates.  They  could  also  work  on  building  and  maintain¬ 
ing  cost  databases  and  other  tools.  The  junior  analysts  would  receive 
support  for  training  and  development  courses,  such  as  those  offered 
through  SCEA.  The  sharing  of  knowledge  among  analysts,  particu¬ 
larly  the  transmittal  of  learning  from  senior-level  to  junior-level  ana¬ 
lysts  would  start  here.11 

Mid-level  analysts  would  be  deployed  to  SPOs  to  work  on  bud¬ 
gets  and  other  day-to-day  cost-estimating  tasks  required  by  the  SPOs. 
They  would  be  SPO  resources  for  a  couple  of  tours,  reporting  to  the 
PD  but  under  the  watchful  eye  of  the  FM  cost  department.  While  at 
the  SPOs,  the  analysts  would  work  closely  with  technical  and  program 
experts  to  gain  in-depth  knowledge  of  specific  space  systems  and  tech¬ 
nologies.  They  would  also  work  with  analysts  from  the  central  cost 


11  Learning  within  organizations  is  a  rich  and  complex  topic  to  which  Linda  Argote’s  book, 
Organizational  Learning:  Creating,  Retaining  and  Transferring  Knowledge,  Norwell,  Mass.: 
Kluwer  Academic  Publishers,  1999,  offers  an  excellent  introduction. 
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organization  in  support  of  special  estimating  requirements,  including 
milestone  reviews. 

After  a  few  years  working  at  the  SPOs  and  gaining  specific  pro¬ 
gram  and  technical  experience,  analysts  would  rotate  back  to  the  cost 
staff  as  senior  analysts.  The  central  cost  organization  should  have 
enough  high-ranking  (GS-13  through  GS-15)  slots  to  attract  and  retain 
these  analysts.  They  would  be  responsible  for  working  on  and  manag¬ 
ing  major  cost  estimates,  such  as  milestone  reviews,  and  would  be  sent 
to  the  SPOs  as  necessary  to  support  this  kind  of  work.  They  would  be 
responsible  for  training  junior  analysts  and  for  working  with  mid-level 
analysts  while  doing  the  major  reviews.  They  would  help  decide  FM 
policy  on  such  issues  as  appropriate  data  collection  and  best  practices 
in  estimating  tools  and  techniques.  A  few  of  the  most  expert  senior- 
level  analysts  could  be  “senior  cost  managers.”  They  would  help  run  the 
system  as  a  whole  and  also  be  in  line  for  senior  leadership  positions. 

The  central  cost  organization  would  be  taking  an  active  role  in 
managing  the  career  progression  of  its  analysts,  providing  opportuni¬ 
ties  for  training  and  growth  in  both  the  central  cost  organization  and 
at  the  program  offices.  However,  the  cost  group  may  also  encounter 
times  where  there  is  a  greater  demand  for  its  resource — skilled  cost 
analysts — than  there  is  adequate  supply  to  meet.  What  happens  in 
these  cases?  SMC  will  need  to  develop  some  kind  of  organizational 
process  to  deal  with  them.  For  other  resource  allocation  decisions,  ana¬ 
lysts  have  proposed  an  additional  “integration”  function  that  stands 
above  the  sources  of  supply  and  demand.  However,  in  SMC  only  the 
very  top  leadership  stands  above  both  the  program  offices  and  the  pro¬ 
posed  cost  group.  We  do  not  suggest  that  SMC  top  leaders  should 
be  engaged  in  managing  these  concerns.  We  do  suggest  that  the  cost 
group  and  the  program  offices  set  up  a  steering  group  that  meets  peri¬ 
odically  (quarterly,  or  as  needed)  to  set  policy  on  assignments  and  to 
develop  an  agreed-upon  strategy  for  managing  cases  where  disagree¬ 
ments  about  perceived  needs  and  availability  of  analysts  arise.12 


12  Leslie  Lewis,  James  A.  Coggin,  C.  Robert  Roll,  The  United  States  Special  Operations  Com¬ 
mand  Resource  Management  Process:  An  Application  of  the  Strategy-to-Tasks  Framework,  Santa 
Monica,  Calif.:  RAND  Corporation,  MR-445-A/SOCOM,  1994. 
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This  approach  provides  a  clear  mechanism  for  career  progression 
and  skill  development.  But  as  with  the  change  in  organizational  struc¬ 
ture  outlined  above,  it  would  be  impossible  without  sustained  senior 
leadership  support  for  change. 

Working  More  Effectively  with  Other  Organizations 

Another  important  concern  is  whether  the  cost  department  can  work 
effectively  with  other  organizations.  There  are  at  least  two  major  classes 
of  organizations  that  cost  analysts  should  be  able  to  interact  with  regu¬ 
larly.  The  first  group  is  the  technical  staff  within  SMC.  The  second 
group  is  other  cost  organizations,  particularly  those  working  on  space 
issues. 

Technical  resources  inside  SMC,  including  the  Chief  Engineer’s 
office,  could  be  a  useful  resource  in  a  number  of  ways.  Cost  analysts 
could  research  technical  questions  in  their  library  (if  this  resource 
exists)  or  by  asking  the  engineers  in  the  Chief  Engineer’s  office.  Cost 
analysts  could  benefit  from  any  technical  training  courses  or  seminars. 
And  cost  analysts  and  engineers  could  benefit  from  temporary  training 
assignments  in  each  other’s  organization.  The  leadership  of  SMC  would 
be  needed  to  support  these  initiatives  by  helping  the  cost  department 
develop  and  maintain  the  linkages  required  for  this  kind  of  productive 
exchange  of  ideas.  We  also  recommend  that  the  SMC  Chief  Engineer 
coordinate  on  all  CARDs  produced  for  milestone  reviews  so  that  the 
SMC  commander  can  obtain  independent  expertise  on  the  program¬ 
matic,  technical,  and  schedule  underpinnings  of  each  program. 

There  are  also  cost  organizations  outside  of  SMC  with  which 
interaction  could  be  mutually  beneficial,  from  sharing  data  and  tools, 
to  sharing  advice,  ideas,  and  lessons  learned  for  current  and  future  pro¬ 
grams.  The  most  obvious  example  is  the  NRO.  While  there  are  some 
links  between  the  two,  our  interviews  revealed  some  constraints  to  an 
open  sharing  of  some  kinds  of  information.  Some  of  this  is  due  to  the 
sensitive  nature  of  the  data,  but  there  are  other  types  of  information 
(e.g.,  lessons  learned,  tool  development)  that  could  provide  a  benefit  to 
both  sides  if  there  were  more  formal  linkages.  Again,  SMC  leadership, 
along  with  the  leaders  of  the  external  organizations,  would  need  to  pro¬ 
vide  the  appropriate  support. 
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Senior  Leadership  Support  Needed  for  Organizational 
Change 

This  chapter  has  outlined  some  suggestions  for  far-reaching  changes 
for  SMC’s  cost  capability.  We  have  identified  changes  to  the  orga¬ 
nizational  structure  that  will  increase  the  independence  of  the  cost¬ 
estimating  function  and  in  our  view  will  increase  the  quality  of  the 
cost  estimates  by  somewhat  insulating  the  cost  estimators  from  pro¬ 
gram  pressure  and  creating  a  formalized  approach  to  career  develop¬ 
ment  and  training.  However,  moving  from  the  current  situation  to  the 
organization  that  we  have  laid  out  in  this  monograph  will  require  a 
significant  amount  of  effort.  Moore  et  al.  offer  a  useful  summary  of  one 
approach  to  organizational  change.13.  Preparation  for  change  includes 
developing  a  case  for  change,  garnering  senior  leadership  support  that 
may  include  a  guiding  coalition,  and  developing  an  action  plan  for 
change.  Support  for  change  includes  sustained  communication,  ensur¬ 
ing  the  right  kind  of  training  and  skills,  providing  incentives,  and  pro¬ 
viding  the  necessary  level  of  resources.  Execution  of  change  involves 
testing  and  validation,  then  full  deployment,  followed  by  monitoring 
and  refinement. 

The  case  for  changing  the  SMC  cost  function  is  clear.  Programs 
have  vastly  overrun  their  initial  budgets,  drawing  the  attention  of  top 
military  leadership — and  of  Congress.  Stronger  and  more  objective 
cost  estimates  would  reduce  the  risks  of  cost  overruns,  particularly 
if  the  FM  cost  department  was  able  to  limit  the  ability  of  program 
leaders  to  shape  an  estimate  that  will  result  in  one  they  think  will  be 
funded.  Senior  leadership  support  is  necessary  for  the  changes  outlined 
above,  and  links  to  the  need  for  resources  and  incentives.  Since  the 
change  involves  multiple  organizations  within  SMC  as  well  as  linkages 
between  SMC  and  external  organizations,  the  commander  of  SMC 
needs  to  be  involved.  SMC/FM  can  develop  an  action  plan  for  change, 
based  on  the  recommendations  provided  here.  FM  needs  to  provide  a 
consistent  message  about  the  change,  and  also  oversee  the  changes.  It 


13  Nancy  Y.  Moore,  Laura  H.  Baldwin,  Frank  A.  Camm,  and  Cynthia  R.  Cook,  Implement- 
ing  Best  Purchasing  and  Supply  Management  Practices:  Lessons  from  Innovative  Commercial 
Firms ,  Santa  Monica,  Calif.:  RAND  Corporation,  DB-334-AF,  2002. 
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can  also  make  sure  the  right  kind  of  training  and  skills  are  available. 
And  it  can  monitor  and  refine  the  new  organizational  structure  and 
associated  changes  to  make  sure  that  it  is  operating  as  intended. 

Daft  suggests  that  successful  change  has  five  elements:  ideas,  need, 
adoption,  implementation,  and  resources.14  The  present  monograph  lays 
out  the  ideas  for  change  and  describes  the  need.  Adoption  occurs  only 
when  key  decisionmakers  decide  to  enact  the  change,  which  is  why 
senior  leadership  support  is  so  important.  Implementation  comes  when 
the  new  ideas  are  in  use  and  may  be  the  most  difficult  step  because 
without  implementation,  no  change  occurs.  Finally,  resources — time, 
attention,  and  investments — are  required  for  successful  change,  which 
again  argues  for  senior  leadership  support.  The  question  of  resources 
is  of  particular  concern  in  a  constrained  budget  environment,  but 
much  of  what  we  suggest  involves  reallocating  existing  personnel  and 
improving  processes  rather  than  hiring  many  new  workers  or  making 
investments  in  expensive  new  technologies.  Ffowever,  it  is  not  possible 
to  accurately  predict  the  cost  of  these  changes. 


Summary 

In  this  chapter,  we  have  examined  a  number  of  issues  concerning  orga¬ 
nizational  structure  and  the  link  between  structure  and  organizational 
performance.  We  offered  an  alternative  organizing  scheme  and  laid  out 
the  benefits  of  making  the  change,  as  well  as  the  challenges  that  SMC 
will  face.  Although  this  change  will  be  difficult,  it  is  nevertheless  nec¬ 
essary  in  order  to  get  improved  performance  from  the  cost-estimating 
function. 


14  Daft.  1998,  pp.  292-293. 


CHAPTER  SEVEN 

Conclusions  and  Recommendations 


Compared  with  other  weapon  systems,  space  systems  have  experienced 
high  cost  growth  during  their  acquisition  phases,  especially  in  recent 
years.  As  a  result,  the  commanders  of  AFSPC  and  SMC  asked  RAND 
Project  AIR  FORCE  to  assess  requirements  and  capabilities  of  SMC 
cost-estimating  organizations,  resources,  tools,  methods,  and  processes 
and  recommend  an  enhanced  approach  for  cost  analysis  aimed  at 
improving  cost  estimation  for  space  systems  and  increasing  the  under¬ 
standing  of  factors  that  influence  their  cost. 

First,  we  focused  on  using  a  RAND-developed  SAR  categori¬ 
zation  methodology  to  identify  the  magnitude  and  causes  of  positive 
cost  variances  on  SBIRS-High  and  GPS  from  1996  through  2005.  The 
results  of  this  research  were  the  following: 

•  The  SBIRS-High  acquisition  program  experienced  substantial  net 
positive  cost  variance,  the  vast  bulk  of  which  can  be  attributed  to 
cost-estimating  errors. 

•  While  the  GPS  acquisition  program  overall  experienced  a  net 
negative  cost  variance,  significant  positive  cost  growth  was  identi¬ 
fied  in  key  components,  particularly  on  the  GPS  IIF  SV  and  on 
OCS.  While  much  of  this  was  due  to  increases  in  requirements 
and  scope,  about  half  was  attributable  to  cost-estimating  errors. 

We  also  noted  that  our  case  study  programs,  both  of  which  were 
formally  launched  in  1996,  got  under  way  during  a  period  of  radi¬ 
cal  transformation  in  both  the  defense  space  industrial  base  and  gov¬ 
ernment  acquisition  policy.  Some  of  the  most  important  changes  that 
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affected  the  ability  of  the  cost  analysis  community  to  produce  credible 
cost  estimates  included 

•  industry  downsizing  and  consolidation  and  increased  competi¬ 
tion  for  fewer  programs 

•  increased  complexity  of  space  systems 

•  implementation  of  acquisition  reform  measures  and  downsizing 
of  the  acquisition  workforce. 

While  these  environmental  factors  were  beyond  the  control  of 
acquisition  officials  and  would  have  posed  special  challenges  no  matter 
how  they  were  approached,  a  variety  of  policy  decisions  and  issues 
directly  related  to  how  SMC  conducted  cost  estimates  made  this  envi¬ 
ronment  even  more  demanding.  Although  many  factors  can  be  singled 
out  as  contributing  to  cost-estimating  errors,  we  focused  on  three  broad 
categories  of  issues  that  our  research  revealed  were  key  causal  factors  on 
the  SBIRS-ffigh  and  GPS  case  studies: 

•  The  cost-estimating  process  appears  to  have  been  organizationally 
too  closely  associated  with  bureaucratic  interests  that  held  advo¬ 
cacy  positions,  making  independent,  disinterested  cost  analysis 
more  challenging. 

•  TSPR  and  other  acquisition  reform  measures  transferred  design 
and  developmental  responsibility  to  contractors  while  greatly 
reducing  the  government’s  ability  to  assess,  monitor,  and  oversee 
contractor  efforts. 

•  Inadequate  cost-estimating  and  risk-assessment  methods  and 
models  were  used  in  both  programs. 

Second,  we  interviewed  personnel  from  all  12  of  the  major  pro¬ 
gram  offices  and  concluded  that  SMC  cost  estimators  must  be  able  to 
render  independent  estimates  on  the  likely  costs  of  acquisition  pro¬ 
grams  without  feeling  corporate  pressure  to  minimize  those  costs  and 
the  inherent  risks  associated  with  them  to  meet  preconceived  notions 
asserted  by  other  components  of  the  acquisition  process.  Further,  TSPR 
and  other  acquisition  reform  measures  from  the  1990s  that  postulate 
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savings  without  underlying  proof  should  either  be  abandoned  or  their 
implementation  approach  be  reconsidered,  because  as  implemented 
they  inhibit  government  oversight  of  contractor  performance  and  pre¬ 
vent  the  collection  of  needed  cost  and  technical  risk  data.  Also,  SMC 
cost  analysts  have  insufficient  cost  and  technical  risk  data  on  new  tech¬ 
nologies  and  subsystems  now  being  incorporated  into  advanced  mili¬ 
tary  space  systems.  This  problem  could  be  ameliorated  by  greater  shar¬ 
ing  of  information,  cost  data,  cost  models,  and  lessons  learned  among 
SPOs,  other  USAF,  and  other  space  procurement  organizations. 

Third,  we  accounted  for  the  cost  analysis  workforce  supply  and 
workload  demands  at  SMC.  In  aggregate  numbers  of  analysts,  con¬ 
tractors  (who  retain  the  bulk  of  the  institutional  memory)  dominate 
the  current  SMC  workforce.  However,  at  least  one  experienced  civil 
service  analyst  heads  each  SPO.  And  while  there  are  fewer  civil  service 
personnel  than  contractors,  both  groups  have  similar  experience  pro¬ 
files.  Unfortunately,  military  analysts  have  limited  career  field  oppor¬ 
tunities  and  do  not  accumulate  experience  because  of  limited  assign¬ 
ments  in  the  cost-analysis  function.  In  comparison  with  the  number  of 
analysts  reported  in  our  workforce  supply  survey,  our  demand  survey 
indicates  that  SMC  is  currently  undermanned  by  about  10  percent. 
However,  modeling  the  workload  and  workforce  available  and  allow¬ 
ing  the  workforce  to  be  freely  reassigned  within  SMC  show  that  the 
workforce  supply  continues  to  be  adequate  to  meet  projected  demands, 
except  during  one  peak  period  in  2007. 

Fourth,  we  assessed  the  methods  and  tools  used  in  cost  estimation 
at  SMC.  We  concluded  that  most  cost  models  used  at  SMC  are  devel¬ 
oped  as  needed  using  a  combination  of  available  databases  and  locally 
collected  data.  As  far  as  we  can  determine,  no  formal  process  is  in  place 
to  capture  locally  developed  data.  The  Unmanned  Space  Vehicle  Cost 
Model  (USCM)  is  the  most  mature  of  the  data-based  models  used  at 
SMC.  The  current  version  addresses  only  spacecraft  bus  and  commu¬ 
nications  payload  costs,  so  other  methods  must  be  used  for  other  types 
of  payloads  and  ground  segment  costs.  A  variety  of  other  models  are 
suitable  in  varying  degrees  for  use  in  particular  circumstances. 

Finally,  we  examined  a  number  of  issues  about  organizational 
structure  and  the  link  between  structure  and  organizational  perfor- 
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mance.  We  offered  an  alternative  organizing  scheme  and  laid  out  the 
benefits  of  making  the  change,  as  well  as  the  challenges  that  SMC  will 
face.  We  cautioned  that  this  change  will  be  difficult,  yet  necessary,  to 
gain  improved  performance  from  the  cost-estimating  function. 

Our  specific  recommendations  are  as  follows: 

•  Institute  independent  technical  and  schedule  reviews.  These  reviews 
should  be  done  in  conjunction  with  cost  estimates  for  major 
reviews  and  milestones.  A  mechanism  or  process  should  be  cre¬ 
ated  for  cost  analysts  to  use  broader  SMC  technical  expertise  as 
a  resource  for  objective  and  independent  technical  and  schedule 
assessments.  We  recommend  that  the  Chief  Engineer  be  required 
to  review  and  coordinate  programmatic,  technical-baseline,  and 
schedule  assumptions  found  in  the  Cost  Analysis  Requirements 
Descriptions  prepared  as  part  of  major  milestone  reviews.  Many 
factors  encourage  optimistic  programmatic  assumptions  and 
foster  low  initial  estimates.  SMC  must  have  long-term  organi¬ 
zational  accountability  not  only  for  cost  estimates,  but  also  for 
technical,  schedule,  and  risk  assessments.  Long-term  customer 
insistence  on  accurate  and  objective  cost  estimates,  at  all  levels  of 
the  Air  Force,  is  critical. 

•  Place  special  attention  on  technical  risk  assessment.  Good  cost  esti¬ 
mates  rely  on  accurate  technical  inputs.  Independent,  rigorous 
formal  technical  risk  assessments  are  needed  to  support  all  cost 
estimates  and  should  be  routinely  updated.  All  cost  and  technical 
risk  assessments  should  be  cross-checked  using  alternative  meth¬ 
odologies  (e.g.,  historical  analogies  compared  with  parametric 
analyses).  The  quality  of  the  inputs  to  the  technical  assessments 
should  be  improved  by  collecting  and  making  available  more  rel¬ 
evant  data  and  increasing  visibility  into  contractor’s  capabilities. 
The  level  of  technical  expertise  and  the  communications  among 
technical,  program,  and  cost  experts  should  be  enhanced. 

•  Adopt  a  hybrid  cost  organizational  structure.  We  reviewed  both 
centralized  and  decentralized  organizations.  In  our  view,  a  hybrid 
structure  that  includes  the  strengths  of  both  centralized  and 
decentralized  structures  has  the  most  potential  benefits  and  the 
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fewest  limitations.  In  particular,  increasing  the  objectivity  of  the 
analysts  who  perform  major  cost  estimates  will  improve  the  reli¬ 
ability  of  the  estimates  and  SMC’s  reputation  as  an  organization 
whose  cost  numbers  can  be  trusted.  Making  this  kind  of  change 
will  require  significant  support  from  senior  SMC  leadership,  as 
we  discuss  below. 

Realign  and  strengthen  the  future  FM  organization  by  assigning 
cost-estimating  tasks  as  recommended.  There  are  specific  tasks  that 
should  be  performed  by  the  cost  staff  within  the  comptroller’s 
office  and  others  that  need  to  continue  being  performed  by  the 
program  office.  Cost-estimating  tasks  should  be  conducted  within 
the  SPO  when  the  focus  is  on  program  execution,  where  changing 
priorities  or  rapid  response  are  common,  for  functions  required 
to  manage  the  day-to-day  activities  of  the  program,  where  the 
official  position  for  effective  interaction  with  SPO  personnel  is 
needed,  and  where  processes  are  unique  to  the  program.  Tasks 
should  be  performed  in  the  cost  staff  when  nonadvocate  analysis 
is  a  priority;  when  experienced  government  leadership,  economies 
of  scale,  and  flexibility  in  assignments  is  desired;  when  skill  sets 
and  tasks  are  outside  SPO  mission;  and  when  workload  and  pri¬ 
orities  are  generally  predictable. 

Require  major  estimates  to  be  led  by  experienced  and  qualified  gov¬ 
ernment  analysts.  Major  cost  estimates  should  not  be  led  by  con¬ 
tractor  support  staff.  However,  contractor  support  does  play  an 
important  role  in  data  collection,  building  cost  models,  docu¬ 
menting  the  results,  and  other  technical  assistance.  The  SMC  and 
other  USAF  human  resources  functions  will  need  to  support  the 
new  staffing  approach.  New  hires,  personnel  assignments,  civil 
service  grade  structure,  and  military  force  development  regula¬ 
tions  may  need  to  be  reassessed  to  attract  and  retain  competent 
cost  analysts  to  SMC.  A  few  experienced  analysts  can  be  more 
effective  than  many  inexperienced  ones. 

Implement  best  practices  from  other  cost  organizations.  Our  team 
held  discussions  with  various  organizations  performing  cost  anal¬ 
ysis  and  identified  best  practices.  Our  interviewees  overwhelm- 
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ingly  agreed  that  sound  initial  estimates  are  critical  and  should  be 
appropriately  resourced.  Other  best  practices  consist  of 

-  including  analysts  with  technical/engineering,  financial/ 
business  management,  economics,  mathematics  and  statistics 
educational  backgrounds  in  cost-estimating  teams 

-  conducting  annual  program  cost  and  risk  estimate  updates 

-  keeping  a  track  record  of  estimates 

-  reviewing  and  archiving  all  major  estimates 

-  emphasizing  monthly  earned  value  management  (EVM) 
analysis  as  a  management  tool. 

•  Standardize  cost-data  collection  and  improve  current  databases.  In 
addition  to  the  historical  cost  information,  SMC  cost  function 
should  also  collect  historical  programmatic,  technical,  and  sched¬ 
ule  data  and  archive  them  for  future  use.  We  encourage  regu¬ 
lar  data  exchanges  with  internal  Air  Force  organizations,  such  as 
AFCAA,  and  external  organizations,  such  as  NRO  and  NASA. 


APPENDIX  A 

RAND  Questionnaire  Used  in  Interviews  with  All 
SMC  Major  Program  Offices,  January  2006 


This  appendix  includes  the  questions  we  provided  to  the  SMC  program 
offices  and  cost  analysis  organization. 


Questions  for  SMC  Program  Offices 

To  be  completed  prior  to  RAND/SMC  staff  visit  to  the  SPO.  Please 
ensure  all  information  in  Part  I  is  unclassified. 

PART  I 

1.  Program  name 

2.  PM’s  name 

3.  POC  at  SPO 

4.  POC  at  AFSPC  for  program 

5.  Is  the  program  an  MDAP? 

6.  When  was  the  last  Milestone  or  major  review? 

7.  What  are  the  upcoming  Milestones  or  major  reviews 
scheduled? 

8.  When  was  the  last  baseline  cost/schedule/risk  (C/S/R)  estimate 
prepared  and  date? 

9.  What  were  the  results  by  year  (TY  and  FY) 

10.  What  is  the  number  of  analysts  assigned  or  matrixed  to  the  SPO 
who  are  involved  in  cost/schedule  or  risk  estimates  or  earned 
value  analysis? 
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a.  Full  time  and  part  time? 

b.  Military,  civil  service  or  contractor? 

11.  How  often  are  C/S/R  estimates  reviewed?  By  whom? 

12.  How  often  is  EV  data  analyzed  and  briefed?  To  whom? 

13.  What  is  your  estimate  of  time/manpower  to  prepare  a  full  life- 
cycle  cost  C/S/R  estimate  for  a  Milestone  or  other  major  pro¬ 
gram  review?  Have  you  collected  actual  data  on  this? 

14.  What  is  your  estimate  of  the  EY  manpower  workload  in  the 
SPO?  Have  you  collected  actual  data  on  this? 

15.  What  technical  support  is  available  to  support  C/S/R  estimat¬ 
ing  and  EV  activities? 

a.  Is  it  organic  or  contractor? 

PART  II 

Items  for  discussion  during  RAND  SMC  staff  visits  to  SPOs — answers 
to  these  questions  are  on  a  nonattribution  basis.  Classified  or  proprietary 
data  may  be  discussed  or  presented  but  should  be  clearly  identified. 

1.  How  are  estimates  presented  and  who  gets  briefed? 

a.  Are  ranges  of  costs  and  risk  analyses  addressed? 

b.  Is  anyone  at  AFSPC  briefed  for  program  reviews  or  on 
annual  Program  Objectives  Memorandum  estimates? 

2.  How  stable  has  program  funding  been  since  the  last  baseline 
C/S/R  estimate? 

3.  How  stable  have  program  technical  activities  been  since  the  last 
baseline  C/S/R  estimate? 

4.  How  stable  has  the  program  schedule  been  since  the  last  base¬ 
line  C/S/R  estimate? 

5.  How  stable  have  program  requirements  been  since  the  last  base¬ 
line  C/S/R  estimate? 

6.  How  can  the  accuracy  of  estimates  be  improved? 

7.  What  steps  do  you  see  in  the  C/S/R  and  EY  processes  which  are 
not  value  added? 

8.  Are  additional  tools  needed  to  perform  your  job  that  you  don’t 
have  (models,  databases)? 

9.  Is  there  sufficient  manpower  to  accomplish  your  required 
tasks? 
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Question  Sent  to  Other  Cost-Estimating  Organizations 

1.  What  is  the  most  difficult  aspect  of  estimating  aircraft:  airframe, 
engine,  avionics,  software?  What  makes  it  more  difficult? 

2.  Do  you  have  a  formal  training  program  for  your  estimators?  Do 
you  have  a  program  to  give  them  different  experiences  in  esti¬ 
mating  (reassignments)  or  formal  mentorship  arrangements?  Do 
you  recognize/encourage  any  formal  certification  programs? 

3.  From  what  background  do  most  of  your  cost  estimators  come? 
How  do  you  recruit  them? 

4.  How  many  cost  estimators  are  in  your  organization?  Can  you 
give  us  a  rough  breakdown  by  experience  level?  Do  you  also 
have  surge  assets  you  can  use,  such  as  FFRDCs,  SETAs,  etc?  In 
your  view  what  are  the  advantages  and  disadvantages  of  organic 
resources  over  contractor-provided  resources?  FMC  will  send 
charts  with  organization  and  experience/grade  levels. 

5.  What  is  your  organizational  structure  (wiring  diagram)?  What 
organizations  do  you  support  (directly  or  indirectly?) 

6.  How  is  your  organization  funded  (both  staff  and  external 
support)? 

7.  Is  retention  of  estimators  an  issue?  Why  do  they  leave 
generally? 

8.  What  tools,  models,  etc.  does  your  organization  use?  What  are 
the  principal  shortfalls  in  current  tools  and  methodologies? 

9.  How  does  your  organization  archive  its  estimates  after 
completion? 

10.  Do  you  keep  cost  estimates  up-to-date  on  a  regular  basis  or  just 
at  the  next  Milestone,  etc.? 

11.  Do  you  keep  any  data  on  how  accurate  your  estimates  were? 

12.  How  does  your  organization  keep  cost  data  from  legacy  and 
current  systems? 

13.  How  much  involvement  does  your  organization  have  with 
earned  value  efforts  or  data?  Do  you  use  it  in  estimating  current 
or  future  systems? 
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14.  Is  your  organization  tasked  with  developing/evaluating  contrac¬ 
tor  proposal  estimates  and  creating  a  most  probable  government 
estimate? 

15.  How  do  you  estimate  manpower  and  schedule  cost-estimating 
efforts? 

16.  How  many  workyears  of  effort  are  required  for  a  “typical”  DAB 
costing  effort  (people  and  time)?  Does  this  vary  widely  among 
different  systems  or  types  of  estimates/reviews? 

17.  How  do  you  obtain  engineering  and  other  technical  assistance 
for  your  estimates? 

18.  What  process  do  you  use  to  get  approval/review  of  estimates  by 
PMs,  PEOs,  SAEs,  etc.? 

19.  How  do  you  handle  situations  where  other  organizations  or  senior 

management  disagree  with  your  estimates?  Is  it  more  desirable 
to  have  multiple  estimates  presented  to  decisionmakers  or  a  rec¬ 
onciled  “single  best  estimate?” 

20.  Do  you  have  “red  teams,”  independent  cost  estimates,  or  other 
second  opinion  types  of  efforts  as  part  of  your  cost-estimating 
process? 

21.  Do  you  track  a  cost  estimate  through  the  PPBS  process  and  com¬ 

pare  budget  versus  estimated? 

22.  Do  you  have  any  advice  or  lessons  learned  for  someone  setting 
up  a  “clean  sheet  of  paper”  cost-estimating  organization? 


APPENDIX  B 


Findings  from  Other  Interviews 


This  appendix  reviews  the  findings  from  interviews  with  all  major 
SMC  program  offices  that  dealt  with  issues  other  than  those  specif¬ 
ically  supporting  the  separate  case  study  effort  on  SBIRS-High  and 
GPS.  RAND  interviewed  personnel  from  all  12  of  the  major  SMC 
space  program  offices  during  January  2006  to  assess  a  wide  variety  of 
issues  affecting  cost  analysis  function,  processes,  manning,  tools,  and 
so  forth.  A  questionnaire  prepared  by  RAND  was  emailed  to  each  SPO 
by  SMC/FMC  in  advance  of  the  interviews.  The  questions  that  appear 
in  Appendix  A  were  provided  to  each  SPO  in  advance  of  our  visit  to 
frame  our  discussion.  Interviews  lasted  between  one  and  two  hours  and 
were  conducted  with  a  variety  of  cost  analysis,  financial  management, 
engineering,  and  program  management  personnel  from  each  program. 
The  program  offices  visited  were  the  Defense  Metrological  Satellite  Pro¬ 
gram  (DMSP),  Evolved  Expendable  Launch  Vehicle  (EELV),  Advanced 
Extremely  High  Frequency  (AEHF)  satellite,  Wideband  Gapfiller  Sat¬ 
ellite  (WGS),  Transformation  Communications  Satellite  (TSAT),  Air 
Force  Satellite  Control  Network  (AFSCN),  Global  Positioning  System 
(GPS)  HR  and  IIR-M,  GPS  IIF,  GPS  IIIA,  GPS  Operational  Control 
System  (OCS  and  OCX),  Space  Based  Infrared  System-High  (SBIRS- 
High),  and  Space  Radar  (SR).1 


1  Months  after  our  visit  with  the  EELV  program  office,  it  was  incorporated  into  the  Launch 
and  Range  Systems  (LR)  program  office. 
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Summary  of  SPO  Responses 

The  following  tables  present  the  results  of  the  interviews.  The  matrix 
shown  in  Table  B.l  presents  the  objective  data  from  Part  I  of  the  ques¬ 
tionnaire  provided  to  RAND  by  the  individual  programs  shown  in 
Appendix  A.  Not  all  questions  applied  to  every  program;  hence,  some 
of  the  blocks  under  each  category  are  blank.  The  first  column  con¬ 
tains  questions  from  Part  I  of  the  questionnaire.  Some  of  the  data  (for 
example,  the  Program  Manager’s  name)  were  collected  to  facilitate 
follow-on  contact  if  needed  and  are  not  reproduced.  Table  B.l  includes 
only  answers  that  were  deemed  relevant  to  the  RAND  study  research 
objectives. 

To  preserve  the  anonymity  of  the  respondents  who  answered  the 
Part  II  questions,  the  following  subsections  synthesize  the  results  of  the 
interviews  rather  than  attributing  responses  to  individual  SPOs.  The 
responses  are  organized  by  subject  category. 

Reporting  of  Estimates,  Audiences,  and  Content 

Four  of  the  programs’  cost  estimates  had  been  briefed  in  the  recent  past 
at  the  Headquarters  U.S.  Air  Force  (HQ  USAF)  level,  which  would 
have  included  program  management  at  SMC  (PM  and  PEO).  One 
SPO  specifically  mentioned  it  had  briefed  its  EAC  to  SMC  manage¬ 
ment.  The  other  seven  SPOs  either  did  not  address  the  issue  or  had 
not  been  required  to  brief  estimates  outside  of  the  SPO  itself  in  recent 
memory.  Due  to  the  constraints  of  the  PPBS  process  and  the  focus 
on  point  estimates  for  use  with  annual  budgets,  ranges  of  estimates 
were  not  normally  addressed,  although  risk  level  assessments  appeared 
to  be  part  of  the  methodology  for  programs  going  on  for  HQ  USAF 
reviews.  Annual  funding  requirements  for  programs  were  provided  to 
HQAFSPC  as  part  of  the  POM  process,  but  formal  briefings  to  HQ 
AFSPC  audiences  did  not  seem  to  be  a  normal  part  of  the  process. 
Only  one  program  office  reported  a  specific  briefing  to  HQAFSPC. 


Table  B.1 

Matrix  of  Responses  from  SPO  Interviews 


Program 


Part  1  Question 

DMSP 

EELV 

AEHF 

WGS 

TSAT 

AFSCN 

M  DAP/AC  AT 

IC(space)/ 

lll(ground) 

1C 

1 

1 

1 

No -II 

Last  major  review 

-1995  (last  SAR 
in  1997) 

2004 

CDR 2004 

I/ll  2000 

IPR  2004 

N/A 

Upcoming  reviews 

None 

MS  III 

None 

None 

IPR(2) 

2007 

PMR  2006 

Last  C/S/R  estimate 

May,  2003 
(IBR) 

6/2005 

2001 

2005 

2005 

N/A 

Assigned  analysts 
(FT/PT) 

2/3 

8/0 

4.5/0 

3/0 

8.5/0  (later  said 

6  FT) 

2/0 

Cost  reviews/by  whom 

None 

Qtrly  DAES/ 
Annual  SARs 

N/A 

EV  briefed/to  whom 

Monthly/ 

Qtrly  PMRs 

N/A  (FAR 

Part  12) 

Monthly/ 

PM 

No  EV 
reporting 

Monthly/PM 
+  qtrly  PMRs 

Monthly/PD 

Time  required  for  LCC 

N/A 

8-12  mo/10 
personnel 

6  mo/7-10 
personnel 

6  mo/7-10 
personnel 

6  mo/7-10 
personnel 

6  mo 

EV  workload 

8-16  hrs/mo/ 
program 

N/A  (FAR 

Part  12) 

2  shared  FT 

2  shared  FT 

2  shared  FT 

1  FT 

Technical  support 
(organization  or 
contractor) 

N/A — program 
in  O&S  phase 

Both 

Aerospace, 

Tecolote, 

Prime 

Aerospace, 

Tecolote, 

Prime 

Aerospace, 

Tecolote, 

Prime 

2  contractors 
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Table  B.1 — continued 

Program 

Part  1  Question 

GPS  IIR/IIR-M 

GPS  ll-F 

IIIA  GPS-OCS-X 

SBIRS-High 

Space  Radar 

MDAP/ACAT 

1  (HR  no  longer 
a  program) 

1 

1 

Pre-MDAP 
(expect  l-D) 

Last  major  review 

9/04 

Upcoming  reviews 

MS  III  2008 

7/07  (SRR) 

Last  C/S/R  estimate 

7/05 

Assigned  analysts  (FT/PT) 

6/0 

2/0 

8/0 

Cost  reviews/by  whom 

N/A 

PMRB 

EV  briefed/to  whom 

Annually 

Monthly/PM 

Monthly/PM 

Time  required  for  LCC 

1  month 

6-12  mos 

EV  Workload 

1  FT 

Technical  support 
(organization  or 
contractor) 

Aerospace 

Aerospace 

ARINC/Mitre/ 

Aerospace 

Both 

NA  =  not  applicable;  FT  =  full-time;  PT  =  part-time;  SRR  =  system  requirements  review;  PMR  =  program  management  review; 
PMRB  =program  management  review  board. 
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Program  Funding,  Technology,  Schedule,  and  Requirements  Stability 

Table  B.2  presents  the  results  of  the  SPOs’  general  perceptions  of  the  rel¬ 
ative  stability  of  the  funding  and  of  technology,  schedule,  and  require¬ 
ments  for  their  programs.  Many  of  the  “Not  Reported”  (N/R)  scores 
reflect  the  acquisition  phase  of  the  program,  i.e.,  the  end  of  the  acquisi¬ 
tion  cycle  was  viewed  as  inherently  stable  across  the  board  because  little 
work  was  left  to  be  accomplished.  In  other  cases,  where  programs  were 
in  early  development  stages,  some  assessment  of  stability  was  viewed  as 
impossible  given  the  immature  nature  of  the  program.  It  is  interesting 
to  note  that  not  all  programs  viewed  themselves  as  either  entirely  stable 
or  unstable  across  the  board,  as  one  might  expect.  The  program  order 
in  Table  B.2  does  not  correlate  with  the  program  order  in  Table  B.l. 

Improving  the  Accuracy  of  Cost  Estimates 

SPOs  in  general  did  not  offer  many  specific  ideas  on  how  to  improve 
the  accuracy  of  cost  estimates.  A  few  mentioned  that  better  access  to 
data  was  needed,  especially  in  those  programs  that  had  been  managed 


Table  B.2 

SMC  Program  Office  Responses  to  Questions  Regarding 
Funding,  Technical,  Schedule,  and  Requirements  Stability 


Program 

Number 

Funding 

Stability 

Technical 

Stability 

Schedule 

Stability 

Requirements 

Stability 

1 

N/R 

N/R 

N/R 

N/R 

2 

S 

S 

U 

S 

3 

U 

U 

U 

S 

4 

S 

S 

U 

S 

5 

U 

u 

U 

N/R 

6 

N/R 

N/R 

N/R 

N/R 

7 

S 

S 

S 

S 

8 

S 

N/R 

u 

u 

9 

N/R 

N/R 

N/R 

N/R 

10 

U 

N/R 

U 

U 

11 

N/R 

N/R 

U 

S 

12 

U 

U 

U 

U 

N/R  =  none 

reported;  S 

=  stable;  U  = 

unstable. 
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under  the  TSPR  and  price-based  acquisition  concepts,  where  little  data 
was  required  from  contractors.  Inability  to  collect  actual  costs  from 
contractors  was  deemed  a  problem  for  government  cost  estimates.  One 
other  item  mentioned  several  times  was  the  need  for  more  cost  data  on 
new  technologies,  which  was  deemed  one  of  the  highest  priorities  for 
cost-data  collection. 

Activities  That  Added  No  Value 

Oversight  by  groups  outside  the  PEO/SAE  reporting  chain  was  named 
most  often  as  an  activity  that  did  not  enhance  the  quality  of  cost  esti¬ 
mates  and  consumed  time  that  the  analysts  could  have  better  spent  on 
their  programs.  In  addition,  one  program  noted  that  several  people  were 
routinely  diverted  from  their  normal  tasks  by  the  need  to  answer  ques¬ 
tions  from  outside  review  agencies,  including  congressional  staffers. 

Need  for  Additional  Tools 

The  SPOs  in  general  did  not  identify  significant  shortcomings  regard¬ 
ing  tools  and  models  but  did  mention  data  as  their  number  one  prior¬ 
ity  area.  Not  only  was  better  access  to  current  contractor  data  required, 
but  also  data  on  new  technologies  were  mentioned  as  a  pressing  need. 

Adequacy  of  Staffing 

Although  selected  overtime  was  expected  for  high  workload  peaks, 
most  SPOs  seemed  satisfied  with  the  level  of  staffing.  This  was  prob¬ 
ably  the  result  of  the  SPOs’  organic  workforce  having  access  to  aug¬ 
mentation  from  support  contractor  personnel  to  satisfy  cost-estimating 
requirements. 


Summary  of  Key  Points  from  SMC  Program  Office 
Interviews 

This  section  highlights  and  summarizes  the  key  points  from  the  12 
interviews  conducted  as  part  of  the  study.  The  overall  issues  are  from 
the  discussion  of  the  questionnaire  topics  as  well  as  other  information 
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that  arose  during  what  were  often  wide-ranging  discussions  between 
RAND  and  the  program  office  personnel. 

First,  as  shown  in  Table  B.2,  most  program  offices  felt  that 
requirements  had  remained  stable  or  did  not  report  problems  with 
requirements.  This  was  especially  true  in  discussions  about  top-level 
requirements  and  key  performance  parameters.  Most  SPO  analysts 
believed  that  overall  requirements  for  their  programs,  as  set  forth  early 
in  the  programs,  had  not  changed  significantly.  Meeting  those  require¬ 
ments  with  technical  solutions  was  where  most  of  the  requirements 
instability  occurred.  This  was  most  typically  caused  by  misunderstand¬ 
ings  or  underestimations  by  both  government  and  industry  of  what 
was  required  to  meet  the  system-level  requirements.  Some  argued  that, 
in  the  competitive  award  environment,  contractors  were  willing  to 
promise  almost  anything  to  win  the  contract  award  and  only  later,  as 
they  proceeded  into  development,  did  the  difficulties  in  meeting  those 
promises  arise. 

Next,  in  terms  of  lessons  learned,  most  SPO  personnel  had  not 
been  involved  in  a  recent  formal  review  up  to  the  HQUSAF  and  OSD 
levels.  Only  three  programs  (EELV,  SBIRS-High,  and  Space  Radar) 
had  undergone  such  a  formal  review  in  the  past  three  years,  and  two  of 
these  were  for  Nunn-McCurdy  breaches,  not  Defense  Space  Acquisi¬ 
tion  Board  Milestone  decisions.  Thus,  most  SPO  personnel  had  little 
actual  first-hand  knowledge  of  the  rigors  and  workload  required  to 
meet  a  full-blown  review  and  the  cost-estimating  process  and  require¬ 
ments  for  such  reviews. 

Most  SPO  analysts  felt  that  new  technologies  were  not  the  key 
problem  in  their  program,  but  rather  the  integration  of  those  technolo¬ 
gies  into  an  operational  system.  Although  much  attention  has  been 
placed  on  using  technologies  that  are  reasonably  mature  (as  measured 
for  example  on  the  Technology  Readiness  Level  scales),  integration  of 
the  diverse  technologies  and  the  software  required  to  do  so  were  seen 
as  areas  of  major  risk.  However,  most  analysts  maintained  that  there 
was  insufficient  cost  data  available  to  estimate  the  costs  of  many  new 
technologies. 

Although  the  SPOs  generally  believed  they  were  adequately  per¬ 
forming  their  earned  value  and  cost-estimating  tasks,  these  sometimes 
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called  for  overtime  to  meet  peak  workloads,  especially  during  intense 
periods  such  as  those  following  the  declaration  of  a  Nunn-McCurdy 
breach.  In  general,  staffing  levels  were  not  mentioned  as  a  major  prob¬ 
lem,  probably  because  support  contractor  augmentation  was  avail¬ 
able  to  support  the  organic  workforce.  EELY  personnel  mentioned  an 
upcoming  shortage  due  to  the  program’s  transition  from  a  FAR  PART 
12  program  (involving  little  cost  reporting  and  analysis)  into  a  FAR 
PART  15  program,  which  would  require  much  more  cost  analysis  and 
earned  value  effort. 

The  implementation  of  Total  System  Performance  Responsibil¬ 
ity,  in  which  the  DoD  essentially  adopts  a  hands-off  policy  with  the 
contractor,  was  almost  universally  seen  as  a  mistake  given  the  cutting- 
edge  R&D  required  to  develop  space  systems.  With  government  over¬ 
sight  minimized  under  TSPR,  programs  were  often  allowed  to  get  well 
off-track  before  corrective  actions  could  be  implemented.  The  lack  of 
cost  reporting  under  TSPR  was  seen  as  a  key  problem  since  financial 
problems  and  cost  overruns  were  not  detected  early  in  the  development 
phases  of  several  programs. 

In  some  instances,  EV  analysis  was  seen  as  not  being  performed 
effectively  or  not  receiving  enough  management  attention.  Some  of 
the  programs  under  TSPR  did  not  even  have  EV  reporting.  Others 
had  been  awarded  as  fixed-price  contracts,  which  do  not  involve  EY 
reporting.  Some  analysts  suspected  that  the  EV  reporting  baselines 
were  inaccurate  or  even  unused  by  contractors.  In  response  to  this  situ¬ 
ation,  SMC  has  issued  guidance  requiring  monthly  EV  analysis  on  all 
programs  where  reporting  is  required. 

As  stated  previously,  insufficient  access  to  historical  cost  data 
was  seen  as  a  critical  shortfall  by  many  of  the  SPO  participants.  Many 
SPOs  had  gathered  data  on  their  own  programs,  but  sharing  among 
SPOs  was  neither  regularly  required  nor  routinely  accomplished.  Not 
only  does  this  non-coordinated  approach  inhibit  the  cost  estimators 
who  need  the  data,  but  it  can  also  lead  to  duplicative  efforts  in  gather¬ 
ing  the  same  or  similar  data.  In  addition,  the  lack  of  data  on  new  tech¬ 
nologies,  especially  those  related  to  payloads,  was  seen  as  a  problem. 
In  addition,  given  the  limited  data  on  cost,  programmatics,  schedules, 
and  technologies,  cost  estimators  must  have  increasingly  sophisticated 
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backgrounds  in  these  areas  if  they  are  to  make  sense  of  the  relevant 
data  that  are  available. 

Finally,  the  three  major  cost  analysis  organizations  in  DoD  deal¬ 
ing  with  space  (SMC,  AFCAA,  and  NRO)  do  not  have  an  effective 
and  coordinated  working  relationship  to  gather,  normalize,  and  share 
data  across  all  the  space  systems  being  developed  and  deployed.  This 
situation  requires  senior  management  involvement  to  break  down  orga¬ 
nizational  barriers  and  permit  freer  flow  of  data  among  the  many  space 
systems  being  developed.  This  would  help  alleviate  the  concerns  noted 
above  about  the  insufficient  data  available  to  SMC  cost  analysts. 


Overall  Conclusions  from  Interviews 

RAND’s  interviews  with  all  12  of  the  major  program  offices  at  SMC 
seemed  to  confirm  many  of  the  most  important  findings  that  arose 
from  the  detailed  case  study  analysis.  These  might  be  summarized  as 
follows: 

•  SMC  cost  estimators  must  be  able  to  render  independent  esti¬ 
mates  on  the  likely  costs  of  acquisition  programs  without  feeling 
corporate  pressure  to  minimize  those  costs  and  the  inherent  risks 
associated  with  them  to  meet  the  preconceived  notions  asserted 
by  other  components  of  the  acquisition  process.  Senior  manage¬ 
ment  must  foster  this  concept  by  asking  for  frank  opinions  about 
potential  problems  areas,  understanding  their  costs,  and  requir¬ 
ing  documentation  and  tracking  of  them. 

•  TSPR  and  other  acquisition  reform  measures  from  the  1990s  that 
postulate  savings  without  underlying  proof  need  to  be  abandoned 
because  they  inhibit  government  oversight  of  contractor  perfor¬ 
mance  and  prevent  the  collection  of  needed  cost  and  technical 
risk  data. 

•  SMC  cost  analysts  have  insufficient  cost  and  technical  risk  data 
on  new  technologies  and  subsystems  now  being  incorporated  into 
advanced  military  space  systems. 
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SMC  cost  analysts  consider  new  technologies,  especially  issues 
related  to  system  integration  and  the  associated  software  develop¬ 
ment,  as  the  highest  risk  areas  on  current  space  programs. 

Greater  sharing  of  cost  data  and  cost  models  among  SPOs,  the 
Air  Force,  and  other  space  procurement  organizations  would  be 
highly  desirable. 

Overall  manning  levels  were  considered  adequate  in  most  SPOs, 
primarily  due  to  the  availability  of  SETA  support  contractors  to 
augment  organic  cost  analysis  capabilities. 

While  we  noted  that  a  more  centralized,  independent  cost  analy¬ 
sis  organization  at  SMC  could  help  alleviate  some  of  these  prob¬ 
lems,  most  SPO  representatives  opposed  losing  control  over  the 
cost  and  technology  risk- assessment  processes. 


APPENDIX  C 


Summary  of  Programs  Currently  Managed 
at  SMC 


This  appendix  provides  a  condensed  overview  of  the  programs  man¬ 
aged  at  SMC  during  the  course  of  this  project. 


Advanced  Extremely  High  Frequency  System 

The  Advanced  Extremely  High  Frequency  (AEHF)  System  is  a  joint 
service  satellite  communications  system  that  provides  near-worldwide, 
secure,  survivable,  and  jam-resistant  communications  for  high-priority 
military  ground,  sea,  and  air  assets. 

Type:  Satellite 

Contractor:  Lockheed  Martin  Space  Systems  Company 

Status:  Satellites  ordered;  first  launch  scheduled  2008 


Advanced  Research  and  Global  Observation  Satellite 
(ARGOS) 

ARGOS  is  an  R&D  satellite  carrying  an  ion  propulsion  experiment, 
ionospheric  instruments,  a  space  dust  experiment,  a  high-temperature 
semiconductor  experiment,  and  the  Naval  Research  Laboratory’s  hard 
X-ray  astronomy  detectors  for  X-ray  binary  star  timing  observations. 
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Type:  Satellite 

Primary  contractor:  Boeing 

Status:  Operational 


The  Defense  Meteorological  Satellite  Program 

The  mission  of  the  Defense  Meteorological  Satellite  Program  (DMSP) 
is  to  generate  terrestrial  and  space  weather  data  for  operational  forces 
worldwide.  Currently  orbiting  satellites  include  the  F-12  through 
F-16.  The  program  is  conducted  in  conjunction  with  NASA  and  the 
National  Oceanic  and  Atmospheric  Administration  (NOAA),  part  of 
the  Department  of  Commerce.  Data  are  furnished  to  the  civil  service 
community  through  NOAA. 

Type:  Satellite 

Primary  contractor:  Lockheed  Martin  Space  Systems 

Status:  Operational;  final  launch  in  2011 


Defense  Satellite  Communications  Systems 

As  the  backbone  of  the  U.S.  military’s  global  satellite  communications 
capabilities,  the  Defense  Satellite  Communications  System  (DSCS) 
constellation  provides  nuclear-hardened,  anti-jam,  high  data  rate,  long- 
haul  communications  to  users  worldwide.  The  system  is  used  for  high- 
priority  communication,  such  as  the  exchange  of  wartime  information 
between  defense  officials  and  battlefield  commanders.  The  military  also 
uses  DSCS  III  to  transmit  space  operations  and  early-warning  data  to 
various  systems  and  users. 

Type:  Satellite 

Contractor:  Lockheed  Martin  Space  Systems  Company 

Status:  Operational 
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Defense  Support  Program 

The  Air  Force  Defense  Support  Program  (DSP)  satellites  orbit  the 
earth  approximately  35,780  kilometers  over  the  equator.  DSP  satellites 
use  infrared  sensors  to  detect  heat  from  missile  and  booster  plumes 
against  the  earth’s  background.  The  DSP  constellation  is  operated 
from  the  Space  Based  Infrared  Systems  (SBIRS)  Mission  Control  Sta¬ 
tion  (MCS)  at  Buckley  Air  Force  Base,  Colo. 

Type:  Satellite 

Contractor  team:  Northrop  Gruman  Space  Technology 
and  Northrop-Grumman  Electronic 
Systems 

Status:  Operational 


Interim  Polar  System 

The  Interim  Polar  System  (IPS)  program  element  provides  protected 
communications  (anti-jam,  anti-scintillation,  and  low-probability-of- 
intercept)  for  tactical  users  in  the  north  polar  region. 

Type:  Satellite 

Primary  contractor:  Boeing 

Status:  Operational 


Milstar 

Milstar  is  a  joint  service  satellite  communications  system  that  provides 
secure,  survivable,  jam-resistant,  worldwide  communications  to  meet 
essential  wartime  requirements  for  high-priority  military  users.  The 
multisatellite  constellation  will  link  command  authorities  with  a  wide 
variety  of  resources,  including  ships,  submarines,  aircraft,  and  ground 
stations. 
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Type:  Satellite 

Primary  contractor:  Lockheed  Martin 

Status:  Operational 


NAVSTAR  Global  Positioning  System 

The  Navstar  GPS  Joint  Program  Office  (JPO)  is  a  joint  service  effort 
directed  by  the  U.S.  Air  Force  and  managed  at  SMC,  Air  Force  Space 
Command,  Los  Angeles  Air  Force  Base,  California.  The  JPO  is  the 
DoD  acquisition  office  for  developing  and  producing  GPS  satellites, 
ground  systems,  and  military  user  equipment.  The  system  includes  sev¬ 
eral  generations  of  satellites:  the  II/IIA  (leaving  service),  the  HR,  IIR-M 
(entering  service),  and  IIF. 

Type:  Satellite 

Primary  contractor:  Lockheed  Martin 

Status:  Operational 


Space  Based  Infrared  System  (SBIRS) 

The  SBIRS  constellation  supports  missile  warning,  missile  defense, 
technical  intelligence,  and  battle  space  characterization.  It  consists  of  a 
system  of  satellites  in  highly-elliptical  earth  orbit  (FIEO)  and  geosyn¬ 
chronous  earth  orbit  (GEO).  The  program  consists  of  several  compo¬ 
nents,  including  the  Mission  Control  Station  (operational  since  2001), 
SBIRS-High  (GEO  satellites  and  HEO  payloads). 

Satellite 

Lockheed  Martin 

Partially  operational  (2  of  4  satellites  in 
orbit) 


Type: 

Primary  contractor: 
Status: 
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Space  Radar  (SR) 

Space  Radar  is  designed  to  give  ground  commanders  of  all  services  an 
eye-in-the-sky  view  of  what  is  on  the  ground  around  them  or  over  a 
mountain  top. 

Type:  Satellite 

Contractors:  Lockheed  Martin  and  Northrop  Grumman 

Status:  Concept  definition.  Demonstrators  to  be 

launched  in  2008 


Transformational  Satellite  Communications  System 
(TSAT) 

TSAT  will  provide  unprecedented  satellite  communications  with  Inter¬ 
net-like  capability  that  will  extend  the  DoD  Global  Information  Grid 
(GIG)  to  deployed  users  worldwide  and  deliver  an  order-of-magnitude 
increase  in  capacity. 

Type:  Satellite 

Primary  contractor:  None 

Status:  Systems  definition  and  risk  reduction 


Wideband  Gapfiller  Satellite 

The  purpose  of  the  Wideband  Gapfiller  Satellite  (WGS)  project  is  to 
provide  flexible,  high-capacity  communications  for  the  military.  WGS 
will  provide  essential  communications  services  for  combatant  com¬ 
manders  to  command  and  control  their  tactical  forces.  Tactical  forces 
will  rely  on  WGS  to  provide  high-capacity  connectivity  into  the  terres¬ 
trial  portion  of  the  Defense  Information  Systems  Network. 
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Type: 

Primary  contractor: 
Status: 


Satellite 

Boeing  Satellite  Systems 

First  launch  was  scheduled  for  June 


2007 


Delta  II 

Delta  II  is  a  medium-lift  launch  vehicle  and  the  workhorse  of  the  Delta 
family. 

Type:  Launch  vehicle 

Primary  contractor:  Boeing  Company  Expendable  Launch 

Systems 

Status:  Operational 


Evolved  Expendable  Launch  Vehicle 

The  EELV  is  a  next-generation  launch  program  designed  to  replace  the 
existing  fleet  of  launch  systems  with  two  families  of  launch  vehicles, 
each  using  common  components  and  a  common  infrastructure.  The 
vehicles  are  the  Boeing  Delta  IV  and  Lockheed  Martin  Atlas  V. 

Type:  Launch  Vehicle 

Primary  contractor:  Lockheed  Martin  and  Boeing 

Status:  Operational 

Inertial  Upper  Stage 

The  Inertial  Upper  Stage  rocket  motor  gives  the  U.S.  government  the 
ability  to  place  satellites  of  up  to  5,300  pounds  into  geosynchronous 
orbit  and  8,000  pounds  out  of  Earth’s  gravitational  field  using  the  Air 
Force  Titan  IVB  rocket  or  NASA  Space  Shuttle. 
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Type:  Launch  Vehicle  (booster  stage) 

Primary  contractor:  Boeing  Corporation 

Status:  Operational,  but  no  planned  launches 


Minotaur 

The  Minotaur  space  launch  vehicle,  produced  for  SMC  under  the 
orbital/suborbital  program,  provides  a  low-cost,  reliable  solution  for 
launch  services  of  government-sponsored  payloads. 

Type:  Launch  vehicle 

Contractor:  Orbital  Sciences 

Status:  Operational 

Minotaur  IV 

Minotaur  IV  is  a  heavier-lift  version  of  the  Minotaur  currently 
under  development  by  Orbital  Sciences.  First  launch  is  scheduled  for 
2008. 

Type:  Launch  Vehicle 

Contractor:  Orbital  Sciences 

Status:  Operational 


Quick  Reaction  Launch  Vehicle 

The  Quick  Reaction  Launch  Vehicle  (QRLV)  program  began  in  FY 
2001,  and  is  launching  up  to  eight  suborbital  vehicles  (one  QRLV  per 
year)  until  FY  2008.  In  addition  to  supporting  DoD  operations  and 
exercises,  the  QRLV  launches  will  also  be  used  for  various  experiments, 
ranging  from  measuring  atmospheric  attributes  to  demonstrating  new 
technologies. 
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Type:  Launch  Vehicle 

Contractor:  Orbital  Sciences 

Status:  Operational 


MILSATCOM  Terminal  Program 

The  MILSATCOM  Terminal  Programs  Office  (MTPO)  pro¬ 
vides  SATCOM  terminals  to  combat  forces  of  all  Services.  It  develops, 
acquires,  and  operationally  deploys  communication  terminals  synchro¬ 
nized  to  support  satellite  weapon  system  operations  and  provides  sup¬ 
port  for  16,000  aircraft,  ship,  mobile,  and  fixed-site  terminals. 

Type:  Support  Program 

Status:  Operational 


Satellite  and  Launch  Control  Systems  Program 

The  Satellite  and  Launch  Control  Systems  Program  Office  serves  as 
the  Air  Force  Satellite  Control  Network  (AFSCN)  acquisition  agency 
responsible  for  network  sustainment  activities;  future  architecture  plan¬ 
ning;  and  data,  communications,  and  range  systems  engineering.  The 
program  office  is  the  primary  interface  to  the  AFSCN  users  for  require¬ 
ments  identification  and  implementation.  This  program  office  is  also 
responsible  for  the  major  development  efforts  of  the  Spacelift  Range 
System  (SLRS).  The  SLRS  consists  of  ground-based  surveillance,  nav¬ 
igation,  communications,  and  weather  assets  centered  at  Patrick  Air 
Force  Base,  Fla.,  and  Vandenberg  AFB,  Calif.,  used  to  support  space 
launch  missions. 


Type: 

Status: 


Support  Program 
Operational 
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Space  Test  Program 

The  DoD  Space  Test  Program  (STP)  is  chartered  by  the  Office  of  the 
Secretary  of  Defense  to  serve  as  “  .  .  .  the  primary  provider  of  mission 
design,  spacecraft  acquisition,  integration,  launch,  and  on-orbit  opera¬ 
tions  for  DoD’s  most  innovative  space  experiments,  technologies  and 
demonstrations”  and  .  .  the  single  manager  of  all  DoD  payloads  on 
the  Space  Shuttle  and  International  Space  Station.” 

Type:  Support  Program 

Status:  Operational 


Command  and  Control  System-Consolidated 

The  MILSATCOM  Command  and  Control  System-Consolidated 
(CCS-C)  system  provides  integrated  launch  and  on-orbit  command 
and  control  functionality  for  MILSATCOM  satellites  as  the  current 
capability  provided  by  the  Air  Force  Satellite  Control  Network  phases 
out. 


Type:  Command  and  Control 

Primary  contractor:  Integral  Systems 

Status:  Operational 


Global  Broadcast  Service 

The  Global  Broadcast  Service  capitalizes  on  the  popular  commercial 
direct  broadcast  satellite  technology  to  provide  critical  information  to 
the  nation’s  war  fighters.  The  CBS  system  is  a  space-based,  high-data- 
rate  communications  link  for  the  asymmetric  flow  of  information  from 
the  United  States  or  rear  echelon  locations  to  deployed  forces. 
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Type:  Command  and  Control 

Primary  contractor:  Raytheon 

Status:  Operational 


SBIRS  Mission  Control  Station 

The  SBIRS  mission  control  station  (MCS)  centralizes  global  command, 
control,  and  communications  for  strategic  and  tactical  warning  into  a 
single  modern  peacetime  facility. 

Type:  Command  and  control 

Contractor:  None 

Status:  Operational 


APPENDIX  D 


Details  of  the  SMC  Cost  Analyst  Workload  Model 


Overview 

RAND  created  a  dynamic  monthly  forecasting  model  that  estimates 
the  number  of  personnel  required  to  complete  schedule,  cost  analysis, 
budget,  and  EV  tasks  at  SMC,  assuming  that  the  current  supply  of 
cost  analysts  will  remain  constant.  In  the  model,  the  primary  work¬ 
force  drivers  are  major  reviews.  For  each  type  of  major  review  (mile¬ 
stone  review  or  source  selection),  the  required  base-case  set  of  analysts 
with  various  experience  levels  (0-3,  3-8,  and  8+  years  of  experience) 
was  determined  by  the  experience  and  judgment  of  RAND  personnel. 
Two  major  modifications  are  made  to  this  base  case:  (1)  the  scale  of  the 
project  as  measured  by  total  acquisition  cost  and  (2)  the  length  of  time 
before  and  after  a  review  due  date. 

To  this  are  added  secondary  and  tertiary  workforce  drivers.  The 
secondary  drivers  are  demands  for  budget  preparations  and  EV  calcu¬ 
lations  that  each  unit  must  complete  continually,  regardless  of  how 
large  the  demands  are  from  the  primary  workforce  drivers.  The  tertiary 
drivers  are  activities — such  as  performing  “what-if”  scenarios,  data  col¬ 
lection,  and  database  development,  training,  and  other  studies — that 
have  widely  varying  frequency  of  occurrence  and  workload  demands. 
On  top  of  all  these  demands  is  the  recognition  that  a  given  number  of 
personnel  will  be  unavailable  for  a  given  amount  of  time  due  to  illness, 
vacation,  and  family  leave. 

In  the  step-by-step  walkthrough  that  follows,  all  tables  and  fig¬ 
ures  are  excerpted  from  the  RAND  model.  There  are  two  types  of 
inputs:  (1)  program-specific  and  historical  data  that  have  been  vali- 
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dated  with  the  latest  available  estimates  and  (2)  assumptions  based  on 
RAND  team  members’  understanding  of  the  staffing  process  of  a  pro¬ 
gram  office. 


Current  Programs 

The  starting  point  for  the  model  is  a  list  of  all  programs  under  SMC’s 
cost  purview,  as  is  shown  in  Table  D.l.  In  the  “Name”  field  is  the 
relevant  SPO  or  program.  In  the  “Cost  Estimate”  section,  we  list  an 
approximate  acquisition  cost  (in  millions  of  TY$),  and  base  year  of 
those  dollars;  this  will  be  used  to  adjust  upward  the  analyst  demand 
for  larger  programs.  The  final  two  columns  are  Major  Reviews  and 
Source  Selections”  require  the  user  to  enter  end  dates  of  major  program 
reviews.  When  no  major  review  dates  were  provided  to  RAND,  we 
assumed  a  three-year  time  period  in  between  major  reviews.  Additional 
major  reviews  and  hypothetical  programs  could  be  added  to  extend  the 
time  line  of  the  analysis. 


Workforce  Assumptions 

The  workforce  needed — the  number  of  full-time  workers  “at  their 
desks” — for  a  major  review  is  determined  by  using  a  set  of  base  cases 
and  a  set  of  multipliers.  In  the  baseline  workforce  shown  in  Table  D.2, 
we  see  that  the  type  of  personnel  demanded  is  broken  in  three  experi¬ 
ence  bands:  up  to  three  years  of  experience,  three  to  eight  years,  and 
eight  or  more  years. 

Table  D.2  should  be  read  as  follows:  the  model’s  base  case  pos¬ 
tulates  that  milestone  review  requires  two  analysts  with  eight  or  more 
years  of  experience,  one  analyst  with  three  to  eight  years  of  experience, 
and  one  analyst  with  fewer  than  three  years  of  experience  (“1/1/2”  for 
short).  The  numbers  in  the  table  are  a  first  cut,  and  have  not  been  justi¬ 
fied  with  thorough  empirical  analysis. 
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Table  D.1 

Projected  Major  Review  Dates  for  Each  Program 


Cost  Estimate 
(TY  Smillions) 


Program 

Total 

Cost 

Base 

Year 

Major  Reviews 

Source 

Selection 

AEHF 

4,602 

2002 

June 

2010 

May 

2013 

May 

2016 

WGS 

1,456 

2001 

March 

2009 

February 

2012 

March 

2015 

TSAT 

12,000 

2009 

September 

2007 

November 

2007 

GPS  IIR-M 

2,574 

1979 

GPS  IIF 

582 

2006 

GPS  IIIA 

7,000 

2009 

August 

2007 

September 

2008 

September 

2011 

August 

2007 

GPS  OCX 

2,000 

2009 

December 

2008 

December 

2011 

December 

2014 

August 

2007 

SBIRS-High 

7,569 

1995 

March 

2006 

February 

2009 

February 

2012 

Space 

Radar 

34,000 

2015 

LR 

24,856 

1995 

October 

2007 

September 

2010 

Table  D.2 

Baseline  Workforce 


Experience 

Milestone 

Review 

Source 

Selection 

0-3  years 

1 

0.5 

3-8  years 

1 

0.5 

8+  years 

2 

1.0 

Next,  each  base  case  is  multiplied  by  the  appropriate  acquisition 
cost  multiplier  in  Table  D.3.  This  factor  is  intended  as  a  proxy  for  scope 
and  complexity  of  analysis:  as  the  constant-dollar  cost  of  the  program 
increases,  the  model  adjusts  the  number  of  personnel  needed  upward. 
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Table  D.3 

Acquisition  Cost  Multiplier 


Cost  Range 
(FY  06  $M) 

Multiplier 

0-1,000 

1.0 

1,000-5,000 

1.2 

5,000-10,000 

1.4 

10,000-20,000 

1.6 

20,000+ 

1.8 

For  example,  we  see  in  Table  D.l  that  the  cost  of  LR  falls  into  the 
$20-billion-plus  category,  yielding  (from  Table  D.3)  a  multiplier  of 
1.8. 

We  also  see  from  Tables  D.l  and  D.2  that  LR  is  projected  to  be 
undergoing  a  milestone  review  in  October  of  2007, 1  requiring  a  base¬ 
line  1/1/2  requirement  for  analysts.  After  applying  the  multiplier,  we 
have  a  requirement  of  1.8/1. 8/3. 6. 

However,  we  know  that  a  review  team  will  have  variable  efforts 
over  time.  Table  D.4  shows  a  notional  pattern  of  how  analyst  require¬ 
ments  change  from  the  time  the  project  team  forms  until  review  due 
date  and  cool-down.  During  the  formation  of  the  project  team  (216 
days  to  180  days  before  the  due  date),  the  multiplier  is  0.5,  meaning 
that  a  program  with  a  1.8/1. 8/3. 6  workforce  requirement  profile,  like 
the  LR  example  calculated  above,  requires  only  half  that  effort  during 
startup.  However,  from  60  to  30  days  before  the  due  date,  there  is  a 
multiplier  of  1.8,  and  3.2/3. 2/6. 3  analysts  are  required. 

In  addition  to  the  primary  workforce  drivers  (the  work  depend¬ 
ing  on  major  reviews),  there  is  work  to  be  done  for  all  time  periods,  the 
staffing  profile  for  which  is  in  Table  D.5.  For  each  experience  band, 
the  model  determines  the  number  of  personnel  required  for  budget 


1  Note  that  October  2007  was  chosen  arbitrarily  because  no  date  is  currently  available  and 
a  specific  date  is  required  in  the  model.  It  is  possible  to  extend  the  model  by  using  a  probabil¬ 
ity  distribution  of  due  dates.  Although  this  would  show  how  schedule  risk  can  affect  person¬ 
nel  planning,  it  would  provide  unrealistically  smooth  demand  profiles,  making  the  model 
less  useful  for  our  purposes. 
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Table  D.4 

Time  Scheduling  Multiplier 

Number  of  Days 
Before  Task 

Due  Date 

Multiplier 

216-180 

0.50 

180-120 

1.50 

120-60 

1.00 

60-30 

1.75 

30-0 

1.25 

0  to  -32 

0.50 

Table  D.5 

Budget  Preparation  and  EV 
Reporting  Workforce 


Experience 

Band 

Budget 

Earned 

Value 

0-3  years 

0 

0 

3-8  years 

0 

0.55 

8+  years 

1.25 

0.55 

Total 

1.25 

1.10 

preparations,  EV  estimates,  and  completion  of  major  cost  estimates. 
Each  SPO  and  FM  requires  1.25  people  for  budget  preparations  and 
1.1  person  for  EV  work.2 

While  EV  work  is  assumed  to  be  constant  across  all  months, 
budget  preparations  are  assumed  to  be  cyclical;  the  RAND  model  pro¬ 
duces  that  cycle  using  the  average  of  two  triangular  distributions.3  The 
model  takes  a  program’s  annual  requirement  for  a  budget  analyst,  and 
reshuffles  it  across  months  within  the  given  fiscal  year  and  adjoining 
fiscal  years.  In  the  aggregate,  instead  of  interpreting  the  model’s  results 


2  The  values  1.25  and  1.1  were  derived  from  analysis  of  the  detailed  workload  demand 
spreadsheets  described  below. 

3  The  seemingly  arcane  triangular  method  was  chosen  for  use  in  an  earlier  version  of  the 
model  that  had  only  a  single  September  peak  for  budget  work.  Had  a  two-peak  distribution 
been  initially  assumed,  a  less-cumbersome  functional  form  would  have  been  chosen. 
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in  terms  of  the  budget  work  required  of  a  single  analyst  in  any  given 
month,  we  interpret  the  results  in  terms  of  total  number  of  analysts 
required  in  that  month. 

For  the  mathematics  of  the  triangular  distribution  to  work,  the 
value  of  the  distribution  is  required  to  be  0  percent  at  the  starting 
point  and  endpoint,  arbitrarily  chosen  so  that  the  first  triangle  is  Sep¬ 
tember  (the  end  of  the  previous  fiscal  year),  and  December  in  the  next 
fiscal  year.4  The  peak  was  chosen  to  be  September,  the  end  of  the  cur¬ 
rent  fiscal  year.  Once  the  ends  are  tied  down  and  the  peak  chosen,  the 
share  of  work  accrued  to  each  month  is  determined  by  simple  algebra.5 
The  16-month  distribution  is  converted  into  a  calendar  year  by  sum¬ 
ming  the  two  Octobers,  summing  the  two  Novembers,  using  other 
values,6  and  multiplying  by  12.  The  same  procedure  was  performed 
for  a  second  triangle,  with  a  peak  at  March,  and  the  ends  at  the  previ¬ 
ous  March  and  following  June.  These  triangles  are  averaged,  and  the 
pattern  is  then  smoothed  with  a  three-month  moving  average.  The 
demand  for  every  budget  analyst  is  then  represented  by  the  x-marked 
smoothed  line  in  Figure  D.l — which  peaks  in  March  and  September. 
This  line  represents  how  the  model  calculates  the  requirement  for  one 
full-time  budget  analyst:  0.89  of  an  analyst  in  December  and  June,  and 
1.11  analysts  in  August  and  September.  The  contribution  of  this  cycli¬ 
cal  trend  to  overall  workforce  requirements  will  be  seen  below. 


4  Reshuffling  across  fiscal  years  has  the  effect  of  slightly  flattening  the  final  distribution  into 
a  shape  that  better  reflects  the  RAND  team’s  anecdotally  based  view  of  how  budget  work  is 
performed.  Choosing  an  endpoint  further  into  the  adjoining  fiscal  year  has  little  substantive 
effect  on  overall  results. 

5  The  peak  budget  work — the  height  of  the  triangle — is  determined  first.  For  all  triangles, 
area  =  (1/2)  x  base  x  height.  In  this  case,  we  have  1  =  (1/2)  x  15  x  height.  (The  space  between 
the  16  months  creates  a  base  15  units  long,  and  the  area  is  the  work  of  1  analyst).  Hence, 
height  =  2  ■=-  15,  or  13.33  percent.  Note  that  this  is  true  regardless  of  which  month  is  chosen. 
But  the  value  for  each  month  before  and  after  the  peak  does  depend  on  the  peak  month.  The 
value  at  each  month  before  the  peak  is  (13.33  percent  -  0  percent)  -5-  12  =  1.11  percent  smaller 
than  the  one  to  its  immediate  right;  the  value  at  each  month  after  the  peak  is  (13.33  -  0 
percent)  +  3  =  4.44  percent  smaller  than  the  one  to  its  immediate  left. 

6  Budget  personnel  requirements  are  assumed  to  be  doubled  in  years  in  FY  2009,  when 
there  will  be  a  Quadrennial  Defense  Review.  To  implement  this,  the  model  doubles  the 
results  of  triangulation  for  2009. 
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Figure  D.1 

Triangular  Distribution  for  Budget  Preparations 


■  First 
triangle 

1  Second 
triangle 

■  Average 
of  both 

'  After 
smoothing 
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The  tertiary  work  category  is  hard  to  define  comprehensively  and 
even  harder  to  measure.  The  tertiary  inputs  used  in  the  model  are  a 
summary  of  all  other  work — working  on  other  studies  and  reviews, 
meeting  base  concerns,  and  responding  to  outside  inquiries.  The  data 
for  this  other  work  come  from  the  survey  instrument  of  workload 
demand  sent  to  SMC.7 

The  workload  demand  survey  sent  to  all  SPOs  was  an  Excel  spread¬ 
sheet  listing  dozens  of  typical  tasks  a  SPO  might  undertake.  For  each 
task,  the  manager  was  requested  to  list  the  number  of  people  assigned 
to  a  task,  the  frequency  with  which  they  are  assigned  to  perform  it,  and 
length  of  time  it  takes  for  them  to  complete  it.  Although  the  survey 
was  meant  to  be  comprehensive,  several  respondents  added  tasks  they 
believed  were  not  clearly  subsumed  in  other  categories. 


7  The  days  unrelated  to  major  reviews  and  training  were  summed,  divided  by  the  total 
workload  demand,  and  multiplied  by  260. 
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A  sample  of  tasks  included  in  the  tertiary  work  category  include 
“Coordination  with  contractors  and  Govt,  data  reviews,  program 
office  training,”  “Serve  as  Team  Leads  and  Team  Members  on  Source 
Selection  Boards,”  “Participate  in  BPA  Qualification  Reviews,”  and 
“Develop/improve  databases,  models,  methodologies,  including  data 
collections.”8 

To  estimate  tertiary  work,  the  RAND  model  schedules  the  tasks 
to  be  completed  and  determines  the  number  of  people  required  to 
complete  them  by  applying  a  set  of  rules  consistently  across  tasks  and 
offices.  It  utilizes  office-unique  expectations  about  the  frequency  and 
duration  of  tasks,  but  averages  the  manpower  required  to  complete 
each  task  occurrence  over  all  offices. 

Essentially,  for  each  office,  we  want  to  know  in  which  months  a 
task  is  performed.  The  model  takes  (1)  the  frequency  of  each  task,  and 
(2)  the  length  of  each  task  and  randomly  assigns  a  start  month  for  each 
task.  For  instance,  if  a  SPO  indicated  that  a  task  occurred  annually, 
a  start  month  was  chosen  randomly.9  For  example,  say  Task  I  starts 
in  December,  and  takes  six  weeks  to  complete;  it  would  have  a  “1”  in 
December,  and  a  “0.5”  for  January.  The  rest  of  the  months  are  “0.”  This 
scheduling  was  done  for  all  tasks,  creating  an  annual  profile  of  tertiary 
tasks  that  was  assumed  to  be  identical  from  one  year  to  the  next. 

Next,  we  calculated  the  median  person-months  per  task  across 
offices10  by  dividing  the  total  person-months  by  annual  frequency 
and  taking  the  median.  The  median  person-months  per  task  was  then 


8  However,  the  tasks  “Develop  cost/schedule/risk  estimates  for  program  initiation/major 
reviews,”  “Develop  cost/schedule/risk  estimates  for  PEO  independent  cost  assessments,”  and 
“Estimate  “what  if”  options”  were  categorized  by  RAND  as  analysis  tasks  directly  related  to 
program  execution.  “Participate  in  Integrated  Baseline  Reviews  (IBRs),”  and  “Maintain  and 
analyze  EV  data”  were  categorized  as  EV  tasks  related  to  program  execution. 

9  If  an  SPO  indicated  a  task  occurred  semiannually,  one  start  month  was  chosen  randomly, 
and  the  other  was  assumed  to  be  six  months  later.  If  an  SPO  indicated  a  task  occurred  three 
times  a  year,  one  start  month  was  chosen  randomly,  the  second  was  assumed  to  be  four 
months  later,  and  the  third  was  assumed  to  be  four  months  earlier,  and  so  on. 

10  Several  tasks  had  to  be  broken  out  into  low  and  high  workload  categories,  because  the 
offices  had  very  different  interpretations  of  the  type  of  work  that  should  be  included  in  the 
task.  The  medians  of  each  subtask  were  then  applied  in  the  model. 
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spread  across  each  task  occurrence  uniformly.  To  continue  with  the 
Task  A  example  above,  say  it  took  nine  median  person-months  to  com¬ 
plete.  Those  nine  person-months  would  be  allocated  across  each  task 
occurrence,  so  that  [9/(1  +  0.5)]  x  1  =  6  people  are  needed  in  December 
and  [9/(1  +  0.5)]  x  0.5  =  3  people  are  needed  in  January  to  complete 
Task  A. 

This  was  performed  for  all  occurrences  of  all  tasks.  The  number  of 
people  needed  in  the  office  to  complete  all  requisite  tertiary  tasks  on  any 
given  month  was  determined  by  summing  across  all  tasks.  The  RAND 
model  then  biters  these  tasks  into  12  major  functions:  major  review, 
estimating,  training,  post-production,  reviews  and  reports,  schedule, 
contract  support,  data,  external,  local,  earned  value,  budget,  and  other. 
Some  tasks  were  broken  equally  into  two  or  three  functions.  Since  the 
schedule  of  source  selection,  major  review,  budget,  and  earned  value 
functions  is  estimated  separately,  their  modeling  results  were  separated. 
In  addition,  the  schedule,  external,  estimating,  and  other  functions 
were  adjusted  for  complexity  using  the  factors  in  Table  D.3. 

It  is  important  to  understand  that  the  workload  data  are  the  com¬ 
bination  of  two  sets  of  judgments:  (1)  those  of  respondents  who  created 
detailed  workload  demand  schedules  referencing  limited  historical  data 
and  (2)  those  of  RAND  personnel  who  categorized  tasks  and  aggre¬ 
gated  them  across  SPOs.  Respondents  might  have  listed  a  smaller  or 
larger  number  of  personnel  than  they  actually  need  (the  errors  could  go 
either  way;  they  are  variances,  not  biases).  Also,  different  respondents 
might  categorize  work  differently,  making  intra-SPO  comparisons  dif- 
bcult  to  interpret. 

Full-Time  Equivalents 

Now  that  we  have  determined  the  number  of  people  required  to 
complete  work,  we  have  to  ensure  that  the  model  estimates  full-time 
equivalent  persons  (FTEs).  From  the  survey  design,  tertiary  work¬ 
loads  are  already  in  FTEs,  but  primary  and  secondary  workloads  must 
be  adjusted  for  periods  of  unavailability.  Days  unavailable  consist  of 
training,  holidays,  family  and  sick  leave,  and  vacation.  Table  D.6  shows 
that,  out  of  a  260-day  work  year,  approximately  41-51  days  (16-19  per¬ 
cent)  of  all  workdays  are  spent  either  on  personal  time  or  in  training. 
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For  an  organization  to  get  a  full  260  days  of  work  from  analysts  with 
fewer  than  three  years  of  experience,  one  needs  1/(1  -  16  percent)  = 
1.19  analysts  on  the  payroll.  This  is  how  the  model  converts  workload 
demand  into  a  requirement  for  workers:  At  any  given  point,  a  demand 
for  ten  workers  on  the  job  is  a  requirement  for  12  on  the  payroll.11 

Now  that  all  workload  demand  has  been  specified  and  periods  of 
unavailability  have  been  taken  into  account,  the  model  calculates  the 
staffing  requirement.  Table  D.7  illustrates  this  with  arithmetic  aver¬ 
ages  over  the  size,  time,  and  FTE  multipliers.  The  result  shows  the 
average  number  of  analysts  needed  to  be  assigned  to  a  SPO — outside 
of  secondary  or  tertiary  demand  for  personnel,  at  each  month,  from 
January  2006  to  December  2010.  (See  Figure  D.2.)  The  bottom  two 

Table  D.6 

Number  of  Days  and  Percentage  of  Time  Unavailable 


Total 

Days  Percent  FTE 

Days  Unavailable  Unavailable  Multiplier 

Full-time  work  year:  260 
Holidays  10 

Family  5 

Sick  13 

Total  28 

Vacation 


0-3  years 

13.0 

(+28) 

41.0 

16 

1.19 

3-8  years 

19.5 

(+28) 

47.5 

18 

1.22 

8+  years 

22.9 

(+28) 

50.9 

19 

1.24 

11  This  assessment  of  unavailable  time  is  realistic  in  terms  of  the  number  of  days,  but  it  could 
be  objected  that  periods  of  training  and  vacation  will  only  be  taken  in  periods  of  relative 
slack.  This  objection  is  valid,  but  sensitivity  analysis  of  the  model  to  account  for  this  showed 
little  practical  difference  in  the  overall  workload  demand— except  that  peaks  were  slightly 
lower  and  valleys  were  imperceptibly  higher. 
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Table  D.7 

Primary  Workload  Demands  and  Multipliers  Yield  a 
Staffing  Requirement 


Milestone 

Review 

Source 

Selection 

0-3  years 

1 

0.5 

3-8  years 

1 

0.5 

8+  years 

2 

1.0 

Total 

4 

2.0 

Average  Size  Multiplier 

1.40 

1.40 

Average  Time  Multiplier 

1.10 

1.10 

Average  FTE  Multiplier 

1.22 

1.22 

0-3  years 

1.8 

0.9 

3-8  years 

1.8 

0.9 

8+  years 

3.7 

1.8 

Staffing  Requirement 

7.4 

3.7 

areas  represent  earned  value  and  budget  drills;  the  tan  area  on  top  rep¬ 
resents  manpower  required  to  complete  major  reviews.  All  areas  in 
between  represent  other  activities  at  each  SPO  and  FM. 

Figure  D.3  represents  the  aggregate  view  of  the  cost  analysis 
workforce  profile  in  the  recommended  organizational  structure — that 
is,  when  analysts  are  allowed  to  move  from  one  SPO  to  another  SPO 
based  on  need.  This  chart  compares  the  demand  for  analysts  as  deter¬ 
mined  in  the  model  versus  the  supply  recently  reported  to  RAND.  On 
the  bottom  is  a  constant  base  of  EY  calculations  and  reporting.  The  next 
layer  up  is  cyclical  budget  document  preparations.  The  “mountainous” 
regions  are  periods  of  major  review,  and  the  “valleys”  in  between  are 
work  on  everything  else.  The  dotted  line  is  at  116,  the  current  work¬ 
force  supply. 
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Figure  D.2 

Workload  Patterns  at  Each  SPO  and  FMC 
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Number  of  analysts 
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Figure  D.3 

Cost  Analysis  Workforce  Demand  Output  from  Model 


1  1  Major  review 

■1  Reviews  and  reports 

1  1  Local 

Estimating 

1  1  Schedule 

1  1  Earned  value 

1  1  Other 

■I  Contract  support 

1  1  Budget 

M  1  Training 

I _ |  Data 

preparations 

HU  Post-production 

1  1  External 

Jan  May  Sep  Jan  May  Sep  Jan  May  Sep  Jan  May  Sep  Jan  May  Sep 

2006  2007  2008  2009  2010 


RAND  MG690-D.3 


APPENDIX  E 


Space  Systems  Cost-Estimating  Tools,  Methods, 
and  Data  Assessment 


Methods,  Tools,  and  Data 

At  its  most  fundamental  level,  cost  estimation  involves  using  informa¬ 
tion  from  the  past  to  make  predictions  about  the  cost  of  future  activi¬ 
ties  or  products.  The  accuracy  of  these  predictions  depends  on  many 
factors,  such  as  the  similarity  of  the  historical  data  to  the  planned  prod¬ 
uct;  the  experience  of  the  estimator  in  interpreting  and  applying  avail¬ 
able  data;  and  the  number  of  “unknown  unknowns”  encountered  in 
the  design,  development,  production,  and  operation  of  the  product  or 
system.  In  some  cases,  the  estimator  can  accommodate  for  these  fac¬ 
tors,  but  others  can  be  addressed  only  by  estimating  their  likelihood 
and  their  potential  effects  on  the  project. 


Methodologies 

Estimators  use  various  approaches  to  develop  their  predictions  of  future 
cost.  The  most  robust  methodology  involves  using  a  database  composed 
of  costs  and  characteristics  of  multiple  similar  projects.  From  it  are 
developed  equations  that  calculate  cost  or  some  intermediate  variable 
(e.g.,  functionality  or  size  of  product)  as  a  function  of  various  physical 
and  technical  characteristics.  These  characteristics,  called  parameters 
(or  sometimes  cost  drivers),  give  the  technique  its  name:  parametric 
estimation.  The  cost-estimating  relationships  are  derived  using  statis¬ 
tical  methods,  most  often  a  form  of  regression  analysis.  Their  accu- 
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racy  depends  in  part  on  the  number  of  independent  data  points,  their 
relevance  to  the  system  being  estimated,  the  selection  of  appropri¬ 
ate  parameters,  and  the  variability  in  the  data  itself.  The  accuracy  of 
the  resulting  estimate  also  depends  on  the  analyst’s  understanding  of 
both  past  and  proposed  projects.  When  properly  applied,  a  paramet¬ 
ric  technique  reflects  the  relevant  experience  with  many  projects  and 
provides  quantitative  measures  of  confidence  in  its  derived  relation¬ 
ships.  Although  parametric  models  take  considerable  time  and  effort  to 
develop,  they  can  be  used  in  a  consistent  way  to  develop  estimates  or  to 
easily  illuminate  the  effects  of  varying  cost-driving  parameters. 

If  sufficient  relevant  data  are  not  available  to  develop  CERs,  ana¬ 
lysts  can  use  an  alternative  approach  called  estimating  by  analogy.  This 
technique  involves  careful  comparison  of  the  characteristics  of  a  simi¬ 
lar  known  system,  subsystem,  component,  or  activity  with  those  of 
the  item  to  be  estimated  and  adjustment  of  the  known  systems’  costs 
to  reflect  differences  between  the  two.  The  accuracy  of  this  method 
depends  heavily  on  the  similarity  of  the  analogous  article  and  the  arti¬ 
cle  being  estimated.  While  good  cost  analysts  always  attempt  to  make 
any  adjustments  as  objectively  as  possible,  this  often  requires  consider¬ 
able  experience  because  the  appropriate  amount  of  adjustment  is  rarely 
obvious. 

A  third  method  used  to  develop  cost  estimates  relies  primarily 
on  expert  judgment.  It  is  generally  used  where  there  is  little  available 
or  relevant  historical  data  on  which  to  base  an  estimate.  It  most  often 
occurs  when  the  system  or  component  being  estimated  has  no  closely 
related  historical  precedent.  Estimates  are  developed  by  asking  subject- 
matter  experts  for  their  estimates  of  some  intermediate  characteristic 
(such  as  labor  hours,  weight,  software  line  of  code,  etc.)  and  translat¬ 
ing  them  into  cost.  Usually,  high,  low,  and  most  likely  estimates  are 
solicited  to  develop  a  risk  distribution.  The  quality  of  expert  judgment- 
based  estimates  depends  a  great  deal  on  the  relevant  experience  of  the 
expert(s),  as  well  as  on  how  the  information  is  solicited  and  validated. 
We  discuss  the  use  of  expert  judgment  in  greater  detail  below. 

Another  estimating  approach,  often  considered  a  separate  meth¬ 
odology,  develops  an  estimate  from  a  summation  of  many  detailed 
component  or  process  estimates.  Usually  referred  to  as  bottom-up  or 
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engineering  build-up,  this  technique  is  most  often  used  by  industry  to 
develop  bid  costs.  Its  appeal  to  industry  is  that  the  data  are  developed 
at,  and  can  be  tracked  to,  the  level  at  which  the  work  will  be  per¬ 
formed,  thus  providing  initial  budgets  for  subsequent  management  of 
the  work.  We  do  not  list  it  as  a  separate  methodology  since  its  defining 
characteristic  is  level  of  detail  rather  than  estimating  approach.  The 
individual  components  of  bottom-up  estimates  are  developed  by  para¬ 
metric  relationships,  analogy,  expert  judgment,  or  most  often,  combi¬ 
nations  of  all  three. 

It  should  be  apparent  that  all  of  these  estimating  methodologies 
depend  on  access  to  relevant  historical  cost  and  characteristic  data.  Col¬ 
lecting  and  normalizing  the  appropriate  data  in  a  form  that  is  useful  for 
cost  analysis  is  a  common  problem  most  cost  organizations  face.  Data 
collection  takes  time,  consistent  focus,  and  significant  effort.  Some  pro¬ 
gram  managers  are  reluctant  to  support  an  effort  that  does  not  directly 
and  immediately  benefit  their  program.  Others  challenge  the  relevance 
of  collecting  data  that  show  the  effects  of  past  errors,  when  the  new 
project  is  focused  on  avoiding  just  such  occurrences.  Those  organiza¬ 
tions  with  robust  databases  have  generally  developed  them  with  the 
strong  support  of  senior  management.  The  database  from  which  SMC’s 
Unmanned  Space  Vehicle  Cost  Model  (USCM)  was  developed  is  such 
a  database.  It  is  discussed  in  more  detail  below. 

However,  SMC  faces  some  additional  challenges  in  maintaining  a 
useful  cost  analysis  database.  First,  space  systems  have  fewer  examples 
of  similar  systems,  subsystems,  or  technologies  than  do  other  weapon 
systems.  The  diverse  missions,  designs,  and  technologies  used  in  many 
aspects  of  space  system  development  result  in  few  uses  of  identical 
components  in  relation  to  aviation  or  missile  systems,  for  example.  The 
challenging  operational  environment  of  space  results  in  very  specialized 
components  that  often  have  few  other  applications.  The  small  quanti¬ 
ties  typical  of  most  space  programs  make  it  unlikely  that  suppliers  will 
make  significant  internally  funded  investments  in  production  efficien¬ 
cies.  The  weight  and  performance  constraints  on  space  vehicles  fre¬ 
quently  require  highly  integrated  custom  payloads  with  significant  new 
development  content  and  few  economies  of  scale.  Finally,  the  diverse 
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needs  of  various  user  communities  limit  the  flexibility  of  developers  to 
address  some  of  these  issues. 

Because  good  analogous  data  for  space  applications  may  be  dif¬ 
ficult  to  find,  estimates  may  be  based  on  a  small  number  of  available 
data  points,  projections  of  dissimilar  programs/activities,  or  expert 
judgment  or  contractor-provided  information  with  limited  empirical 
support. 

Since  there  are  circumstances  in  which  space  program  estimators 
have  to  rely  on  expert  judgment,  it  is  worthwhile  to  examine  the  char¬ 
acteristics  of  good  expert  judgment  and  how  to  best  use  it  as  a  source 
of  reliable  information  for  decisionmaking. 

Using  Expert  Judgment 

Software  engineering  researchers  have  been  investigating  the  accuracy 
of  various  cost  and  schedule  estimation  methods  for  many  years.  The 
information  technology  and  cost  estimation  literature  reveals  a  vari¬ 
ety  of  relevant  results.  For  example,  Kitchenham  et  al.  evaluated  185 
projects  that  used  a  combination  of  eight  types  of  estimation.1  By  far, 
expert  judgment  was  the  most  accurate.  This  may  be  true  for  software 
projects;  however,  Morgan  (1981)  and  Ruckelshaus  (1985)  argue  that 
experts  often  disagree  and  are  prone  to  biases  just  as  laypersons.  On 
the  other  hand,  Jorgensen  analyzed  a  large  body  of  project  results  and 
found  the  following  mixed  results:2 

•  In  five  studies  (of  15),  expert  judgment  was  more  accurate. 

•  In  five  studies,  formal  estimation  models  were  more  accurate. 

•  In  five  studies,  there  was  no  difference  in  accuracy  between  formal 
estimation  models  and  expert  judgment. 


1  Barbara  A.  Kitchenham,  Shari  Lawrence  Pfleeger,  Beth  McColl,  and  Susan  Eagan,  “A 
Case  Study  of  Maintenance  Estimation  Accuracy,”  Journal  of  Systems  and  Software  64, 
November  2002. 

2  Magne  Jorgensen,  “Estimation  of  Software  Development  Work  Effort:  Evidence  on 
Expert  Judgment  and  Formal  Models,”  unpublished  draft,  April  7,  2006a;  Magne  Jorgensen, 
“A  Review  of  Studies  on  Expert  Estimation  of  Software  Development  Effort,”  Journal  of  Sys¬ 
tems  and  Software  70(1- 2),  2006b,  pp.  37-60. 


Space  Systems  Cost-Estimating  Tools,  Methods,  and  Data  Assessment  197 


Although  cost  analysts  in  general  consider  formal  models  pref¬ 
erable  to  subjective  judgment,  in  fact  the  body  of  empirical  evidence 
shows  no  clear  superiority,  especially  for  state  of  the  art  projects  where 
little  to  no  data  are  available.3  Jorgensen  suggests  seven  guidelines  for 
using  expert  judgment.4  His  guidelines  are  based  on  actual  experience, 
are  easy  to  implement,  and  are  derived  from  the  most  recent  research 
on  the  subject.  The  last  characteristic  is  particularly  important;  since 
both  software  and  hardware  development  practices  have  changed  over 
the  years,  older  research  may  be  less  relevant. 

Incorporating  Risk 

Risk,  the  likelihood  of  an  unfavorable  outcome  and  uncertainty  is 
the  indefiniteness  in  outcome — favorable  or  unfavorable — will  be  an 
inherent  part  of  every  cost  estimate.  There  are  many  reasons  why  risk 
analysis  should  be  included  in  cost  estimates.  For  instance,  informa¬ 
tion  such  as  technical  requirements  used  in  the  cost  estimate  are  often 
either  ill  defined  or  not  well  understood;  further,  the  economic  condi¬ 
tions  related  to  the  producers  of  technologies  evolve  over  time.  Thanks 
to  increased  emphasis  on  evaluating  and  presenting  risk  as  a  required 
part  of  estimates,  most  decisionmakers  now  understand  that  a  cost  esti¬ 
mate  is  either  a  range  of  numbers  or  actually  a  statistical  distribution  of 
probable  costs  and  that  selecting  an  appropriate  confidence  level  is  one 
of  the  decisions  to  be  made. 

Despite  this  emphasis  and  continuing  research  in  risk  analysis, 
evaluating  and  quantifying  risk  remains  one  of  the  more  difficult  chal¬ 
lenges  in  estimating.  While  statistically  derived  parametric  models  can 
quantify  their  estimating  uncertainty,  the  uncertainty  around  the  pro¬ 
grammatic/technical  inputs  must  be  modeled  separately.  With  other 
estimating  methodologies,  all  the  risk  must  be  modeled  discretely, 


3  j0rgensen,  2006a. 

4  Magne  Jorgensen,  “Practical  Guidelines  For  Expert-Judgment-Based  Software  Effort 
Estimation,”  IEEE  Software  21(3),  May-June  2005. 
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using  both  historical  data  and  consultation  with  experts,  as  described 
previously.5 

In  some  cases  it  may  be  difficult  to  get  assistance  from  qualified 
experts  who  are  not  already  involved  with  the  program.  To  address  this 
issue,  at  least  for  key  decision  point  estimates,  the  Independent  Pro¬ 
gram  Assessment  (IPA)  Team  concept  was  initiated.  These  teams  have 
proven  valuable  in  providing  the  milestone  decision  authority  with 
an  independent  perspective  on  the  risks  faced  by  the  program  under 
review.  Unfortunately  for  the  cost  estimators,  these  IPAs  are  normally 
constituted  only  for  major  program  estimates  and,  even  then,  much 
of  the  cost  estimator’s  work  must  be  done  before  their  assessments  are 
available. 


Estimating  Tools  at  SMC 

Hardware  Models 

As  a  result  of  the  conditions  described  above,  most  models  used  at 
SMC  tend  to  be  developed  on  an  as-needed  basis  for  a  specific  applica¬ 
tion  using  selected  data  points.  Depending  on  their  needs  and  avail¬ 
able  resources,  some  programs  may  also  collect  cost,  technical  and 
programmatic  data  to  augment  the  data  contained  in  published  data¬ 
bases.  Unfortunately,  apparently  no  formal  process  is  in  place  to  ensure 
locally  collected  data  is  added  to  these  databases  and  thus  made  avail¬ 
able  to  other  SMC  organizations.  General-purpose  CERs  are  primar¬ 
ily  used  for  common  components/activities  such  as  systems  engineer¬ 
ing  integration  and  test/program  management  (SEIT/PM).  There  are, 
however,  several  general  purpose  estimating  models  available  for  use  in 
developing  rough  early  stage  or  rapid  turnaround  estimates. 

The  most  developed  of  these  models  is  the  USCM.  SMC  and  its 
predecessor  organizations  have  sponsored  the  development  and  main¬ 
tenance  of  USCM  for  over  three  decades.  The  model  and  database  are 
strongest  on  common  bus  components,  which  tend  to  be  more  evo- 


5  For  additional  information  on  quantifying  risk  in  acquisition  programs  see  Arena,  Leon¬ 
ard,  et  al„  2006;  and  Arena,  Younossi,  et  al.,  2006. 


Space  Systems  Cost-Estimating  Tools,  Methods,  and  Data  Assessment  199 


lutionary  from  program  to  program  than  do  payloads,  which  tend 
to  be  more  specialized.  The  current  version  is  weakest  on  non-com¬ 
munications  payloads  and  non-recurring  costs.  Its  programmatic  and 
schedule  data  are  limited  to  top-level  actuals  with  limited  insight  into 
original  plans  and  changes  made  during  the  course  of  program  execu¬ 
tion.  The  current  USCM  model  (Version  Eight)  and  database  address 
only  the  spacecraft  (and  communications  payload,  if  any)  and  do  not 
include  data  on  ground  segments.  Figure  E.l  shows  the  spacecraft  data 
in  USCM  from  military,  commercial  or  NASA  projects  categorized 
by  decades.  Figure  E.2  shows  the  number  of  observations  included  in 
USM  from  communication  and  passive  sensor  payloads. 

In  recent  years,  the  data  upon  which  the  USCM  model  is  based 
have  been  shared  with  the  National  Reconnaissance  Office  (NRO) 
and  NASA.  Some  attempts  are  under  way  to  integrate  Air  Force  and 
NRO  estimating  tools.  Unfortunately  classification  issues  hamper  data 
exchange  from  the  NRO  to  SMC.  The  USCM  database  is  currently 
being  expanded  and  the  older  data  is  being  renormalized  to  conform  to 
the  new  MIF-HDBK  881A  work  breakdown  structure. 

Figure  E.l 
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Figure  E.2 

Payloads  and  Payload  Components 
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In  addition  to  USCM,  various  other  space  vehicle  models  are 
available  at  SMC  and  are  used  in  appropriate  situations.  Table  E.l 
summarizes  their  key  features. 

Summary 

Three  fundamental  methods  are  used  in  estimating  the  cost  of  future 
space  systems: 

•  Parametric  Method.  The  known  costs  of  a  number  of  similar  arti¬ 
cles  are  mathematically  related  to  the  physical,  technical,  and  pro¬ 
grammatic  characteristics  that  influence  those  costs. 

•  Analogy.  The  known  costs  of  a  particular  article  are  adjusted  for 
its  differences  with  the  article  being  estimated. 

•  Expert  Judgment.  Subject  matter  experts  are  asked  to  character¬ 
ize  the  differences  and  similarities  of  the  article  being  estimated 
with  related  articles  or  efforts  in  their  experience  and  these  rela¬ 
tionships  are  used  to  infer  the  cost  of  the  article  of  interest. 


Table  E.1 

Space  Vehicle  Models  Available  at  SMC 


Model 

Content 

Utility  for  SMC 

Remarks 

Unmanned  Space 
Vehicle  Cost 

Model 

USAF/NASA/commercial 
spacecraft;  communications 
payloads 

Useful  for  ROM  estimates; 
limited  due  to  scope,  age, 
data  documentation 

45  data  points;  many  from  1970s;  no 
non-COMM  payload  or  ground 
segment 

Passive  Sensor 

Cost  Model 

Space  sensor  components 

Limited  due  to  age  and 
quality  of  data 

Older  data;  planned  to  be  updated  and 
incorporated  into  next  USCM  version 

NASA/Air  Force 
Cost  Model 

USAF/NASA;  orbital/interplanetary/ 
manned  spacecraft;  instruments; 
launch  vehicles;  engines 

Special  cases 

122  data  points;  integrated  risk  and 
phasing  capabilities;  heavily  adjusted 
data 

Small  Satellite 

Cost  Model 

Spacecraft  <1000  kg;  orbital/ 
interplanetary 

Special  cases 

35  data  points 

PRICE 

Commercial  general  purpose 
model 

Detailed  comparisons  of 
relative  cost  of  alternatives 

Various  specialized  modules  available 

SEER 

Commercial  general  purpose 
model 

Detailed  comparisons  of 
relative  costs  of  alternatives 

Various  specialized  modules  available 
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All  these  methods  depend  on  access  to  relevant  historical  costs 
and  characteristics  data.  Developing  databases  for  cost  analysis  requires 
a  significant  long-term  commitment  of  effort  and  resources.  Because 
of  the  limitations  in  the  availability  of  relevant  historical  data,  space 
cost  analysts  must  often  work  closely  with  subject  matter  experts  to 
ensure  their  assumptions,  methodologies,  and  assessments  of  risk  are 
appropriate. 

Evaluating  and  quantifying  risk  should  be  an  integral  part  of  any 
cost  estimate.  Using  the  risk  information  provided  by  the  analyst,  the 
decisionmaker  must  select  an  appropriate  confidence  level  for  budget 
formulation  and  approval. 

Most  cost  models  used  at  SMC  are  developed  as  needed  using  a 
combination  of  available  databases  and  locally  collected  data.  As  far 
as  we  can  determine,  no  formal  process  is  in  place  to  capture  locally 
developed  data.  The  USCM  is  the  most  mature  of  the  data-based 
models  used  at  SMC.  The  current  version  addresses  only  spacecraft 
bus  and  communications  payload  costs,  so  other  methods  must  be 
used  for  other  types  of  payloads  and  ground  segment  costs.  A  variety 
of  other  models  are  suitable  in  varying  degrees  for  use  in  particular 
circumstances. 
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